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Executive Summary 


Having shown its legacy training program for low-income young adults to be highly effective 
(Fein & Hamadyk, 2018), Year Up undertook the challenge of adapting this intensive, stand- 
alone model for improved scalability. The resulting adaptation is its Professional Training Corps 
(PTC) program, which maintains key features of the original model but operates on college 
campuses and leverages college instruction and facilities to reduce costs. 


Launched at one college in 2010, the PTC attracted strong interest and quickly spread to other 
locations. By mid-2019, PTC programs were operating on over 15 college campuses and 
serving over 2,000 young adults annually. 


Year Up initially approached the PTC as a pilot initiative, emphasizing incremental, evidence- 
based improvements. In 2015, the organization began expanding the number of PTC sites in 
operation. It nonetheless kept local programs fairly small—generally serving fewer than 150 
participants a year—while continuing to assess outcomes, address ongoing challenges, and 
conduct small impact studies. Once the program is meeting performance targets consistently, 
Year Up anticipates participating in a larger impact study. Its leaders describe their two-part 
approach to evidence-building as “to improve and to prove.” 


As part of its strategy for evidence-based improvement, Year Up partnered with a team of 
researchers at Abt Associates and the University of Pennsylvania to secure financial support for 
its development and improvement work from the Institute of Education Sciences (IES). This 
Executive Summary describes the research goals and main findings of the IES-funded project. 
The team completed fieldwork and data collection in early 2020. Hence, the report does not 
reflect any effects the COVID-19 pandemic may have had on PTC or associated outcomes. 


The Professional Training Corps 


The PTC program model includes all elements of Year Up’s stand-alone core program but alters 
some features to reduce costs. A well-designed and implemented randomized controlled trial 
(RCT) found the core program very successful in increasing participants’ earnings (Fein and 
Hamadyk, 2018). But the core program’s high cost—about $35,000 per participant in 2018— 
makes scaling prohibitive. 


Compared to the core program, PTC programs target a similar population of young adults and 
provide similar services over two six-month phases—learning and development (L&D) and 
internships. The target population is 18-24 year-olds from low-income backgrounds who have a 
high school diploma or equivalent. Local program staff recruit and screen to identify applicants 
who are motivated and face life challenges deemed manageable with the range of supports the 
program provides. 


While preserving the main elements of the core program, the PTC model revises the service 
delivery approach, shifting some instruction responsibilities to colleges and reducing the 
intensity of other services. Like the core program, during the first six months (L&D), participants 
receive four types of services: (1) training in basic academic (mainly English), professional (i-e., 
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“soft”) and technical skills; (2) intensive coaching; (3) close monitoring; and (4) feedback on 
behavior and support from staff and peers within organized learning communities of about 40 
participants. As in the core program, time is reserved for weekly learning community activities 
such as “Friday Feedback” and guest speakers. 


During the second six months, program participants move to full-time internships at area firms, 
where they gain entry-level experience in a target field such as IT or financial services. The 
structure of the internship phase is similar to that in the core program. Participants work full-time 
four-and-a-half days a week and return to the Year Up site each week for a half-day workshop 
called the Internship Seminar. 


As in the Year Up’s core program, participants earn a weekly stipend throughout the year, 
conditional on meeting Year Up’s strict behavior code. Performance standards under both 
program models emphasize full-time work at good wages in Year Up target occupations after 
program completion. Both programs draw heavily on revenue from employer payments for 
interns. 


The main difference is that local PTC programs use college instruction and donated campus 
office space with the goal of reducing cost while maintaining performance comparable to that of 
the core programs. PTC participants take academic and technical courses from instructors at 
the partner colleges (usually as a cohort) and have access to the same college facilities and 
supports as other students at the college. Whereas internship revenue covers about 60 percent 
of the core program’s costs, Year Up’s goal for PTC is to cover costs entirely through employer 
payments. 


Like their counterparts in the core program, PTC program staff provide training in professional 
skills, arrange and monitor internships, provide coaching and other supports, run learning 
communities, and monitor participants’ behavior. However, participants in PTC programs have 
substantially fewer contact hours with Year Up staff than do participants in the core programs 
(about 22, compared to 35, contact hours a week during the L&D phase). Stipends also are 
lower in PTC than in the core program ($50/week during L&D and $150/week during internships 
versus about $150 and $225, respectively, in the core program). 


The PTC financial model is calibrated to reach break-even by reducing the average cost per 
participant to about $20,000 and by generating commensurate revenue from employer 
payments for interns). The model aims to reduce costs primarily by substituting college courses 
for Year Up instruction (thereby reducing staffing costs), by using donated office space on 
campus, and by reducing stipend amounts. Hitting revenue targets requires retaining at least 83 
percent of participants to the revenue-generating internship phase and collecting payments at 
the full rate ($26,000 per intern) for at least 90 percent of interns. Core programs are subject to 
the same revenue targets and generally have met them. 


Study Approach 


The project consisted of two overlapping periods of activity. Work began with in-depth study of 
several PTC program components and then shifted to a more global assessment of overall 
operations and impacts. 
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During the first three years of the project (which began in 2015), the study team conducted 
quick-turnaround “mini-studies” of three high-priority implementation challenges. Each sought to 
identify promising approaches to addressing these challenges and provide preliminary evidence 
on the response of Year Up and its local PTC programs to the study findings. 


In the last two years of the project, the team assessed PTC from a more global perspective. 
One strand of evaluation was an assessment of implementation fidelity. A second strand 
measured impacts on education and employment outcomes through a small RCT in three PTC 
programs. As the length of follow-up possible under the IES grant timeline was very short, the 
goal for this small RCT was to implement an experiment whose outcomes could be followed 
over a longer time period in a subsequent study.' Thus, though this report provides initial impact 
analyses, the findings do not yet provide a firm basis for assessing program success. Finally, 
the team interviewed diverse program stakeholders towards the end of the project period to 
ascertain their perspectives on progress and challenges going forward. 


The study team designed this project to address the major objectives of IES’s Development and 
Innovation (D & |) grant program. Such grants support projects that develop and document 
evidence-based interventions to improve education outcomes. Closely related IES grant 
programs support more rigorous effectiveness and replication studies of interventions 
developed through D & | projects. By funding these complementary types of projects, IES aims 
to support progression across successively higher levels of evidence-building. 


Key Findings: Mini-Studies 


Unlike the typical IES D & | project, this one did not begin with a pre-specified focal intervention. 
Rather, the study team engaged practitioners—including Year Up and its college and employer 
partners—in prioritizing needs and solutions. In this way, the team sought to better identify 
critical improvement needs, mobilize experienced staff to generate more effective solutions, and 
promote ownership of, and commitment to, improvements. 


The study team guided the process through a series of structured discussions to, first, identify a 
wide range of topics of interest and, then, progressively winnow the list to three issues that 
would be addressed in mini-studies. Key selection criteria were that the issue would be of high 
priority to PTC stakeholders, addressable via a relatively short-term study, and not already the 
focus of a major improvement effort. 


Each mini-study used different research methods. In lieu of final written reports, study leaders 
communicated findings to Year Up national and local staff through more informal and explicitly 
interactive means—generally involving a series of PowerPoint briefings via webinars. Year Up’s 
national staff played a key role in soliciting, sharpening, and guiding the program’s responses to 
study findings and recommendations. 


1 A grant from Arnold Ventures to Abt Associates is supporting this longer-term follow-up. 
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Mini-Study #1: Improving Academic Outcomes and Retention 


The first mini-study tackled a central challenge in the core-to-PTC adaptation—establishing 
strong monitoring of participants’ academic performance and providing timely support to those 
experiencing challenges. It used an RCT to test staff-developed improvements in academic 
monitoring and supports during the program’s six-month L&D phase. Over two enrollment 
cycles, the study team randomly assigned 317 PTC enrollees in three local programs to either a 
treatment group (“Improvement Strategies Group”) or a control group (“Usual Strategies 
Group”). 


During cycle 1, program staff used the improvement strategies they had developed for 
monitoring and supporting PTC participants in the treatment group and used their usual 
strategies for participants in the control group. Based on experiences with the improvement 
strategies implemented for the first enrollment cohort, local staff modified their approaches for 
the second cycle. 


The study team monitored the strategies being used with the Improvement Strategies Group 
and, after the two cycles of testing concluded, compared program retention and college 
enrollment rates for the two groups. Overall, the improved monitoring boosted retention to the 
internship phase by 10 percentage points and college persistence by 13 percentage points 
(statistically significant at the 10-percent and five-percent levels, respectively). Impacts on 
program retention and college persistence for cycle 2 were even larger: 14 and 20 percentage 
points, respectively (both statistically significant at the five-percent level). 


A year after the mini-study ended, the three offices reportedly had institutionalized the improved 
coaching practices refined in cycle 2, and national staff were encouraging additional locations to 
adopt the improvements. Local PTC staff cited the mini-study’s approach to collaborative 
development and testing as a best practice worth extending to other improvement needs. 


Mini-Study #2: Setting Up Successful Internships 


The second mini-study assessed strategies for working with employers to develop processes for 
setting up internships that consistently provide high-quality workplace experiences. It drew on 
in-depth interviews and focus groups with current PTC interns and other stakeholders to make 
qualitative assessments of current strategies and identify potential improvements in the 
processes for setting up successful internships. 


Discussions with participants, employers, and program staff indicated that the quality of PTC’s 
internship experiences was uneven. Some companies—particularly larger firms with established 
internship programs—consistently provided stimulating learning opportunities and supportive 
social environments, but others failed to consistently do so. 


A brief literature review helped to identify markers of good internships, including clear learning 
objectives; well-designed skill development activities; opportunities for rotation through different 
types of positions; steps to minimize and otherwise make good use of downtime; and active 
engagement with supervisors and co-workers. 
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Based on the findings, the study team identified steps that Year Up national and local staff could 
take to support employers in setting up high-quality internships. The team made two 
overarching recommendations. The first was for Year Up national and local staff to inventory 
and make accessible a living inventory of firms’ best practices, drawing on Year Up’s deep 
reservoir of internship experience. The second was to create communities of practice among 
employers to foster networking, brainstorming, and sharing of best practices. 


The mini-study also recommended that local staff: (1) create more detailed descriptions of each 
internship so that participants could better prepare in advance; (2) strengthen training and 
planning with employers around the kinds of activities and supports they would provide— 
potentially specifying these as part of written agreements; (3) provide technical assistance to 
employers and participants in developing individualized six-month learning plans for internships; 
and (4) help employers to establish high-quality processes for monitoring and coaching interns. 


Interviews one year after the mini-study found Year Up staff still committed to making the 
recommended improvements. The organization had taken some steps to improve coordination 
with employers around describing internship positions. They also had augmented new manager 
training and had strengthened existing monitoring and feedback mechanisms. More substantial 
improvements recommended in this mini-study remained on Year Up’s “to do” list. 


Mini-Study #3: Fostering College Persistence 


The third mini-study explored the tensions between Year Up’s traditional emphasis on full-time 
employment and increased interest in college completion—tensions that arose from the 
partnerships of local PTC offices with local colleges and increased interest in fostering longer- 
term career advancement. The study team used a mixed-methods approach that combined 
statistical analysis of patterns of college persistence and employment following PTC and core 
program completion with insights from in-depth interviews and focus groups. 


Both the literature and findings from analyses of Year Up data indicated that full-time 
employment substantially reduces the likelihood of college enrollment, considerably lengthens 
the time to degree, and decreases the likelihood of degree receipt. Although about twice as 
many PTC as core graduates attended college in the year following the program, longer-term 
analyses by Year Up show that few in either group earned degrees. 


As context, this mini-study cited external evidence that a college certificate or degree can be 
critical for advancement in many fields. In interviews, a number of Year Up national leaders 
nevertheless questioned the “college for all” thesis, noting that experience and industry 
certifications offered viable alternatives in IT and other occupations. Other leaders affirmed the 
need for college degrees but felt that an intervening spell of work with good earnings was 
important for positioning many young adults to afford college. 


In synthesizing input to arrive at recommended improvements, the team sought to identify ways 
that PTC programs could promote college continuation without weakening post-program 
employment outcomes. Our central recommendation was that Year Up introduce a systematic 
approach to career planning that would balance and further both goals. The recommendation 
envisioned local PTC programs working with participants—as well as college advisors and 
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worksite supervisors—to develop personalized career plans that include specific steps for both 
employment and continuing education. Plans would identify sequences and combinations of 
work and school consistent with each participant’s goals. The plans would provide the basis for 
monitoring of graduates’ progress and local programs’ performance. 


By mid-2019, Year Up had implemented the initial components of a “guided career pathways” 
approach along the lines recommended. These components included new tools, curricula, 
performance goals for Year Up staff, and progress goals for participants. Going forward, careful 
analysis is needed to determine whether the revised approach is identifying solid next steps for 
education and training (as well as employment) and the degree to which Year Up’s young adults 
are taking those steps. 


Key Findings: Implementation Fidelity 


This component of the project assessed the degree to which the PTC program was operating as 
intended at the end of the four-year study period. Year Up’s general goal was for PTC programs 
to meet the same performance targets as the core program—but at a substantially lower cost. 


The study analyzed a series of performance outcomes up to mid-2019. Analyses are based on 
two primary data sources—administrative statistics from Year Up’s management information 
system, and an online survey completed by all 15 PTC local program leads and 14 of their main 
college liaisons. 


The findings show that Year Up had made substantial progress in implementing the PTC 
program by mid-2019. The program achieved high fidelity relative to goals for lowering per- 
participant costs, operating with a leaner staff, recruiting young adults who fit the target profile, 
collecting full payments for internships, and placing high proportions of graduates in well-paying 
jobs or in school. 


The program consistently struggled to meet goals for recruitment and retention. In the 
discussion section below, we summarize recommendations for strengthening recruitment that 
emerged from a mid-2019 round of stakeholder interviews. Concerning retention, Mini-Study #1 
showed that improved academic monitoring and supports could generate a substantial boost in 
retention for a modest cost. 


Recruitment and retention shortfalls are the main reasons why local PTC programs to date have 
generated only three-fifths of the internship revenue needed to fully cover their costs.? Our 
findings suggest strong potential for addressing both shortfalls. 


2 During the same period, internship revenue also covered about three-fifths of the core program’s costs, 
but for different reasons. Due to higher retention rates, the local core programs collected substantially 
more internship revenue per participant, effectively meeting their revenue targets. But because the 
core program’s per-participant costs are substantially higher (about $35,000) than PTC’s ($22,000), 
revenue covered a similar fraction of costs in the two programs. 
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Key Findings: Early Impacts on Education, Employment, and Earnings 


A small RCT designed to measure the overall impacts of the PTC programs had generated only 
a short follow-up period as of the cut-off point for data used in the present report. Recognizing 
that local PTC programs had not yet reached the standards Year Up envisions for full 
implementation, the findings are useful for seeing whether impacts are headed in the right 
direction, but they do not represent the effects of a fully implemented program. Furthermore, as 
discussed earlier and below, only limited follow-up was possible during the period of this IES 
grant. More time will be needed to determine if this RCT will yield promising impacts. 


Over a two-year period beginning in July 2017, staff at three PTC programs enrolled four 
cohorts of program applicants (554 young adults) in the study. Those randomly assigned to the 
treatment group were encouraged to participate in PTC. Those assigned to the control group 
were not allowed to participate in PTC, but they could apply to other education and training 
programs at the college or elsewhere. This report presents very early evidence of program 
impacts, focusing on short-term outcomes for the earliest two cohorts. Analyses draw on 
administrative data on college and employment outcomes from the National Student 
Clearinghouse and the National Directory of New Hires and cover the in-program and 
immediate post-program periods (Qs 1-4 and 5 following enrollment, respectively). Subsequent 
reports will analyze impacts for all four cohorts over a longer follow-up period.* 


Year Up’s local PTC programs produced moderately large increases in college enrollment 
during the program’s L&D phase and smaller increases during its internship phase. Given the 
programs’ close college partnerships—with participants taking college courses and PTC 
programs co-located on campus—one might have expected impacts on college enrollment 
larger than in the core program during the program year (as reported in Fein and Hamadyk, 
2018). But impacts on any enrollment were about the same for the two programs. 


One reason is that the PTC programs in the study served a population of young adults with 
higher rates of college enrollment than did the core programs. Higher enrollment rates in the 
PTC control group set a higher bar for the PTC treatment group to surpass in order to show 
impacts. Another reason is that attrition between random assignment and the start of classes 
was higher in PTC than in the core program, due to delays and uncertainties in the college 
admissions process—especially financial aid determination. 


The PTC programs’ impacts on cumulative months of full-time college enrollment nonetheless 
exceeded those of the core program in every quarter, mainly due to higher rates of full-time 
enrollment among the PTC participant group. By the end of Q5, the PTC treatment group had 
gained 2.8 more months of full-time college enrollment than their control group counterparts. 
The corresponding figure for the core program was 1.6 months. 


The early PTC findings on employment and earnings impacts also mirror those for the core 
program (as reported in Fein and Hamadyk, 2018). During the in-program period (Q1-Q3), the 
PTC treatment group had significantly lower employment rates than the control group 


3 Future reports will be developed under a separate grant from Arnold Ventures. 
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(reductions varying from 10 to 25 percentage points). However, by the Q4, when participants 
were beginning to complete the program, the reduction had shrunk to 7 percentage points. By 
Q5, when none of the treatment group was still in the program, employment rates were similar 
in the treatment and control groups. 


For the full program year (Q1-Q4), the PTC treatment group’s average earnings were 50 
percent below that of the control group ($5,367 versus $12,984). The treatment-control gap of 
$7,617 is about 40 percent larger than was seen for the core program, after adjusting for 
different rates of program participation. 


As in the evaluation of Year Up’s core program, PTC treatment group members’ employment 
and earnings increased substantially as they completed the program. By the time all participants 
had completed the program, the average earnings of the PTC treatment group slightly exceeded 
those of the control group, though the difference was not statistically significant. Longer-term 
follow-up is needed to determine whether larger, statistically significant earnings impacts 
emerge. 


Discussion 


By mid-2019, Year Up had made substantial progress in developing and implementing its PTC 
program model. The required adaptations were not trivial, and launching programs at more than 
15 colleges is a notable feat. Attesting to the model’s viability, PTC programs continue to 
operate at all but one of the colleges where they were launched, and enrollments have grown— 
albeit more slowly than Year Up envisioned. 


Implementation fidelity analyses show that the PTC programs made substantial progress on 
many performance goals. They have yet to consistently meet some key targets—for example, 
recruitment, retention, and revenue. The study team explored the prospects for surmounting 
these and other challenges in a final (Spring 2019) round of phone interviews with stakeholders: 
Year Up national leaders, local PTC leads, liaisons at partner college, and employers. 


Interviewees identified a number of plausible sources of recruitment difficulties. Some causes 
were quite broad—such as declining numbers of 18- to 24-year-olds, plentiful (if not highly 
remunerative) job opportunities in a strong economy, and aggressive competition for students 
from a burgeoning for-profit education sector. PTC-specific challenges included convincing 
young adults that PTC offered a surer route to a good career than traditional college programs, 
and addressing attrition of applicants before the start of classes. 


Stakeholders suggested a variety of responses to recruitment challenges. These responses 
included hiring staff with stronger community ties, formalizing efforts to increase contact 
frequency within and across recruitment cycles, strengthening collaboration with colleges and 
other local entities in developing applicant pipelines, and adjusting the PTC model to 
accommodate a wider target population. 


Year Up and college staff agreed that local arrangements sometimes operated more like “real 
estate deals” than authentic partnerships. Year Up staff believed that greater engagement by 
college staff and faculty in program activities would help to extend and strengthen the PTC 
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learning community and thereby promote higher retention. College liaisons said they would like 
to see PTC staff and students more often at campus events. 


More generally, college liaisons identified a number of motives for partnering with Year Up, 
including boosting enrollment of non-traditional students, increasing completion rates while 
maintaining academic standards, and connecting PTC program graduates to follow-on college 
programs. Colleges also expressed keen interest in learning how to offer Year Up—style 
professional skills training and internships to their students more widely. 


Finally, Year Up, college staff, and employers interviewed agreed that the program should do as 
much as is feasible in one year to promote participants’ longer-term career success. Thinking on 
how to accomplish this evolved somewhat during the study period, due in part to improvements 
in career planning growing from Mini-Study #3. The main advance was to strengthen and 
formalize services to equip graduates with clear long-range plans and next steps for education 
and employment. Relatedly, Year Up committed to monitoring whether graduates were “on 
track” with their plans as part of its one-year post-graduation follow-up. 


At this juncture, several aspects of a robust “guided career pathways” approach remain to be 
fully implemented and assessed. A number of local PTC programs have taken steps to 
strengthen alignment among participants’ career interests, available PTC training tracks, and 
occupational foci of internship opportunities. But more can and should be done to implement the 
team’s recommendations for career planning. One good step would be to do more to involve 
college advisors and internship supervisors in career planning during PTC. 


Perhaps most importantly, while Year Up continues to monitor against program-wide standards 
for employment goals, the new career planning approach largely defers to young adults’ 
preferences on goals for education and training. With such an approach, monitors might find 
that graduates are “on track” with their career plans and yet had made little formal commitment 
to or progress in education and training. Year Up should assess the skills-focused goals in the 
career plans that its new approach is producing and strengthen related guidance as needed. 


Looking Ahead 


This report has documented substantial progress on PTC program implementation to date and 
Year Up’s clear grasp of the remaining challenges. Although the current IES grant has ended, 
there are still many possibilities for valuable mini-studies to help solve these challenges and a 
variety of possible approaches to related research collaborations. 


As of the end of this IES grant, only limited follow-up was available to assess PTC’s impacts 
based on the two small RCTs implemented for this project. Under a separate grant, the study 
team will extend analysis and reporting on impacts over a longer follow-up period. 


The large, ongoing RCT of the core program (Fein and Hamadyk, 2018)—which similarly 
followed years of work by Year Up to perfect implementation and a small pilot RCT—shows the 
value of the “to prove” step in Year Up’s improve-prove paradigm. Once the PTC program 
model has worked out remaining challenges in meeting its performance goals, the study team 
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will be working with Year Up to assess potential approaches to a larger-scale RCT to measure 
its effectiveness. 
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1. Introduction 


Millions of young adults leave high school with high hopes but limited prospects for a successful 
career. Insufficient financial and social capital make it difficult to access, navigate, and complete 
college. Absent a college credential, it has become increasingly difficult to find a well-paying job. 
Failure to take initial education and career steps during young adulthood elevates sharply the 
risk of long-term economic disconnection and its wider societal ramifications (Belfield et al., 
2012; Lewis & Gluskin, 2018; Loprest et al., 2019; National Research Council, 2014; Ross et al., 
2018). 


The last two decades have brought a variety of promising programs addressing these 
challenges. Randomized controlled trials (RCTs) have found that a wide range of education, 
training, and employment strategies can improve outcomes.‘ Most of the programs tested have 
been small, however, and few have fostered success on both the college and employment 
fronts. This report provides early evidence on a next-generation program designed to address 
both domains—Year Up’s Professional Training Corps (PTC) program. 


The PTC program model is an adaptation for college settings of Year Up’s successful stand- 
alone program, which it calls the “core” program. A federally-soonsored RCT conducted as part 
of the Pathways for Advancing Careers and Education (PACE) project found that the core 
program produced large increases in earnings extending to the end of an initial three-year 
follow-up period (Fein & Hamadyk, 2018). Like the core program, PTC programs serve young 
adults aged 18-24 with a high school diploma or equivalent. Both programs provide six months 
of full-time basic skills and technical training and supports followed by six-month internships in 
information technology (IT), financial services, and other high-demand occupational sectors. 


Unlike Year Up’s core program, PTC programs operate on college campuses and use college 
instruction and donated facilities to reduce Year Up’s costs and improve scalability. The core 
program’s earnings gains were not accompanied by improved college outcomes (Fein & 
Hamadyk, 2018). College partnerships heighten both the expectations and possibilities for doing 
more to foster college success. 


Since PTC began in 2010, more than 15 colleges have started PTC programs, and additional 
partnerships are in the pipeline. All locations succeed in putting the program’s major 
components in place and all programs have operated continuously since their inception. As this 
report documents, the PTC program to date has made substantial progress but has not yet 
reached the level of performance Year Up expects. Key performance gaps include shortfalls in 
recruitment and retention and a general need to tighten college partnerships. 


Anticipating challenges, Year Up has maintained a strong emphasis on research and 
improvement throughout its developing of the PTC program. One strategy in its learning agenda 


4 |llustrative are studies of college success strategies such as ASAP (Weiss et al., 2019) and the 
LaGuardia Bridge Program (Kallison, 2017) and of employment-focused sectoral training programs 
such as Per Scholas (Maguire et al., 2010; Schaberg, 2017), Project QUEST (Roder & Elliot, 2019), 
and the Year Up core program (Fein & Hamadyk, 2018). 
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has involved partnering with outside evaluators to conduct research on performance gaps and 
promising program improvements. 


In 2015, the study team for this study—comprising researchers at Abt Associates and the 
University of Pennsylvania—received a four-year grant from the federal Institute of Education 
Sciences (IES) to study the developing PTC initiative. The goals of the project were to identify 
and develop evidence-based responses to high-priority implementation challenges; assess the 
fidelity of local program operations to the PTC model; and mount a small-scale RCT of the 
model in several high-fidelity programs. This report presents findings from the project. 


The team had completed its fieldwork and data collection by early 2020. Hence, the report does 
not reflect any effects the COVID-19 pandemic may have had on the PTC program or 
associated outcomes. 


This introductory chapter provides background on the PTC model and this evaluation. It begins 
with a summary of the PTC model (Section 1.1). Section 1.2 discusses Year Up’s approach to 

launching pilots at a growing number of sites and how dissemination to date attests to the new 
model's viability. Finally, Section 1.3 provides an overview of the study approach. 


1.1 The PTC Program Model 


To understand the PTC program model, it is helpful to provide a brief sketch of the Year Up core 
program on which it is based. This section describes the core model, adaptations for PTC, and 
the PTC financial model. 


1.1.1. The Year Up Core Program 


Refined over nearly two decades, Year Up’s stand-alone core program provides a year of full- 
time education, training, and work experience. It currently serves about 2,000 low-income young 
adults (aged 18-24) annually in nine major cities. Local admissions teams screen applicants for 
motivation and indications they will be able to handle and benefit from the program. 


During the first six months—the learning and development, or L&D, phase—participants receive 
training from Year Up staff in professional and technical skills, help with academic and personal 
challenges, feedback on behavior, and positive reinforcement from organized learning 
communities of staff and peers. The roughly 40 young adults in each learning community take 
classes and participate in weekly activities together. In the second six months—the internship 
phase—the program places students in full-time internships at major firms, where they gain 
entry-level experience in IT, financial services, or other high-demand occupations. 


Year Up provides its core program participants a weekly stipend, conditional on meeting 
standards in a written behavior contract. Stipends average just under $150 a week during the 
L&D phase to $225 a week during the internship phase. Contract infractions trigger reductions 
in stipends. Repeated infractions can reduce stipends to zero, at which point students are 
dismissed from the program. 


A novel aspect of the core program’s financing is that employers make substantial payments to 
Year Up in exchange for hosting interns. In addition to the value of the work that interns do, the 
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internships can provide employers with a cost-effective pipeline for recruiting new hires. 
Revenue from internships covers about 60 percent of Year Up’s costs. Private philanthropy 
covers nearly all of the rest: Only two percent of costs are funded through grants from 
government agencies. 


As noted previously, a large RCT found that Year Up’s core program had high retention and 
produced substantial increases in participants’ earnings (Fein & Hamadyk, 2018). Of young 
adults randomly assigned to the treatment group, 75 percent completed the intensive one-year 
program. Following an initial year of reduced earnings during program, treatment group 
members’ average earnings climbed to levels that remained 40 percent above those of the 
control group over the next two years. The study found large impacts in all subgroups examined 
and in all eight local offices. 


In contrast, the core program had mixed impacts on college persistence. During the first 
(program) year, the practice of co-enrolling participants at local partner colleges boosted 
treatment group members’ college enrollment well above the level in the control group. As 
treatment group members found jobs in Year 2, their college enrollment fell to levels below 
those of the control group. In Year 3, the two groups’ enrollment rates equalized. 


Notwithstanding these generally positive results, Year Up recognized that certain features of the 
core program would make it difficult to reach the scale needed to achieve the organization’s 
mission to close the “Opportunity Divide” for millions of young adults. In Year Up’s analysis, key 
constraints included the core program’s high costs ($28,290 per participant in 2013-2014)° and 
the operational burden involved when a single organization directly provides all of the program’s 
extensive services. 


In response, Year Up began exploring a variety of strategies for improving the scalability of its 
services. These strategies include adaptations preserving the core program’s main features; 
efforts to package and disseminate individual program strategies; and wider campaigns to 
influence attitudes, practices, and policies in ways that encourage adoption of key principles 
underlying Year Up models. The Professional Training Corps program is the largest Year Up 
initiative focused on adapting the core model for improved scalability. 


1.1.2 The Professional Training Corps (PTC) Model 


The PTC program model preserves the core program’s components but shifts some costs to 
colleges and reduces the intensity of some services. Exhibit 1-1 summarizes the key similarities 
and differences between the two models. 


Similarities with Year Up’s core program. Like the core program, PTC begins with a six- 
month L&D phase. During this period, participants receive four types of services: (1) training in 
basic academic (mainly English), professional (i.e., “soft”), and technical skills; (2) intensive 
coaching; (3) close monitoring; and (4) feedback on behavior and support from staff and peers 


5 As noted in Chapter 3 of the present report, by 2018 average costs for the core program had risen to 
around $35,000 per student. 
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organized into learning communities. As in the core program, time is reserved for weekly 
learning community activities such as “Friday Feedback” and guest speakers. 


Exhibit 1-1: Comparison of Year Up’s Core and PTC Programs 


Program Phase Core Program PTC 
Recruitment e Broad community outreach e Generally similar, PTC recruitment mostly 
e Digital advertisement and social media adding to regular college admissions pipeline 
outreach campaigns (versus tapping into it) 
e Multi-step screening process e Close coordination with college admissions 
and registration processes 
Learning & e Six months of full-time training and support e Same basic components as core program 
Development in stand-alone Year Up offices e PTC staff teach professional skills. College 
e Courses in technical, English, and instructors teach technical and English 
professional skills taught by Year Up staff courses 


e College credit for coursework available via e Students earn credit at partner college 
American Council on Education-accredited e Students spend about 22 hours each week in 


curriculum learning community with staff 

e Participants and staff organized into learning | e —Stipends of $50/week, reduced for contract 
communities (35 hours/week) infractions 

e Regular and ad hoc professional 
development activities 


e Substantial coaching and supports provided 
by staff advisors and social workers 

e Contracts with each participant specify 
expected behaviors 

e — Stipends of up to $150/week, reduced for 
contract infractions 


Internship e — Six-month, full-time internship with major e Very similar to core program 
local employers e Stipends of $150/week, reduced for contract 
e Workshops and advising at Year Up one infractions 
afternoon/week 


e Close monitoring and troubleshooting of 
internship experience by Year Up staff 

e — Stipends of up to $220/week, reduced for 

contract infractions 

Up to 12 months of job search and e Same, with somewhat more emphasis on 

placement services after graduation college enrollment in addition to employment 


Post-Program 


4 During the PACE study period (2013-2014) core program participants were eligible for credit through agreements 
with local colleges at each site, rather than through the American Council on Education (ACE)’s Credit 
Recommendation Service, and post-program employment services extended four months after graduation, rather 
than 12 months (Fein & Hamadyk, 2018). 


In the second six months, PTC program participants move to full-time internships at area firms, 
where they gain entry-level experience in areas such as IT and financial services. The structure 
of the internship phase is similar to that in the core program. Participants work full-time four- 
and-a-half days a week and return to the Year Up site each week for a half-day workshop called 
the Internship Seminar. 


As in the Year Up’s core program, participants earn a weekly stipend throughout the year, 
conditional on meeting Year Up’s strict behavior code. Performance standards under both 
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program models emphasize full-time work at good wages in Year Up target occupations. Both 
programs draw heavily on revenue from employer payments for interns. 


Differences between the core and PTC programs. One major difference is that the PTC 
program uses college instruction and donated campus office space with the goal of reducing 
costs while maintaining performance comparable to that of the core program. PTC participants 
take academic and technical courses from instructors at the partner colleges (usually as a 
cohort) and have access to the same college facilities and supports as other college students. 
Year Up staff continue to provide training in professional skills, arrange and monitor internships, 
provide coaching and other supports, run learning communities, and monitor behavior against 
Year Up contracts. 


The PTC model requires effective coordination between local Year Up staff and their college 
partners on recruitment and admissions. It also requires effective instruction in academic and 
technical skills that participants need to succeed in jobs and education following the program. 
Fostering longer-term educational success requires good alignment of the college courses PTC 
participants take with longer programs of study leading to marketable college credentials. 


Course loads for PTC participants are comparable to a full-time college schedule. Because 
these courses are all taught by college instructors rather than Year Up staff, as is the case in 
the core program, PTC participants spend fewer hours each week in Year Up learning 
communities. PTC participants average about 22 hours in learning communities with other 
participants and staff each week during the L&D phase, compared to 35 hours for core program 
participants. 


Stipends are lower in PTC than in the core program. PTC participants receive up to $50/week 
during L&D and $150/week during internships—about one-third and two-thirds of the 
corresponding core program amounts in each phase, respectively. Year Up’s initial reasoning 
for lower stipends was that many PTC participants would receive pocket money from Pell and 
other grants after paying tuition costs. 


Differences between academic and PTC program calendars lead to slight differences in 
programming compared to the core program. For example, academic terms typically are 
somewhat shorter than the six-month L&D phase. PTC programs commonly fill the gap with 
“boot camps” and other training to prepare participants for their internships. 


The PTC program puts more emphasis on securing college credit for internships and on 
participants taking one or more additional courses during internships. In addition to contributing 
to progress towards degree attainment, this emphasis on college credit helps to maintain 
participants’ eligibility for student financial aid. This aid generates additional revenue for Year 
Up’s college partners and helps to make the PTC program financially worthwhile for them. 


Year Up staff provide career and alumni services to PTC program graduates to foster positive 
career outcomes. As in the core program, Year Up’s main goal for participants is full-time 
employment. The PTC program puts somewhat more emphasis on encouraging young adults to 
continue taking college courses while working in full-time jobs after graduation. Later chapters of 
this report explore the relationship between these goals. 
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Year Up’s financial model for PTC assumes a need for fewer Year Up staff than for the core 
program. Participant-to-staff ratios range from 10:1 to 7:1 in PTC program offices, compared to 
about 4:1 in core program offices. 


1.1.3. PTC Financial Model 


Year Up’s core program has demonstrated the possibility of securing substantial revenue from 
employers in the form of internship payments. This revenue covered 59 percent of the core 
program’s average $28,290 per participant cost during the PACE study—an unprecedented 
level of private sector financing for a workforce training program serving low-income youth. 
Employers interviewed for PACE expressed satisfaction with the return on their investments. 
They cited the value of the work that interns performed, a cost-effective pipeline to entry-level 
hires post-internship, and positive feelings from contributing to local communities (Fein & 
Hamadyk, 2018). 


A key financial goal is for PTC programs to produce outcomes comparable with the core 
program while covering costs entirely through revenue from employers’ internship payments. 
The PTC financial model is calibrated to reach break-even by reducing the average cost per 
student to about $20,000 and by generating commensurate internship revenue. 


As noted in the previous section, the PTC model reduces costs by substituting college courses 
for Year Up instruction (thereby reducing staffing), by using donated office space on campus, 
and by reducing stipend amounts. To hit revenue targets, PTC programs must retain 83 percent 
of participants to the revenue-generating internship phase and collect payments at the full rate 
($26,000 per intern) for at least 90 percent of interns. Core offices generally have met these 
revenue targets. 


As discussed in Chapter 3, most local PTC programs have not yet achieved these financial 
targets. Lower than anticipated retention in PTC programs has meant that fewer participants 
have reached internships, and internship revenue thus has fallen short of levels needed to 
break even. 


1.2 Assessing Usability 


An initial research objective in IES Development and Innovation projects is to assess the degree 
to which newly developed interventions are “usable.” Usability is very similar to viability, as 
defined by IES: “the extent to which the intended user understands or can learn how to use the 
intervention effectively and efficiently, [that they are] physically able to use the intervention... 
and willing to use the intervention.” To establish usability, researchers must show that 
interventions are viable in “authentic educational settings.” © 


Year Up’s successful launch of PTC programs at 15 colleges—mostly since 2014—provides 
strong evidence of the model's usability. As summarized in Exhibit 1-2, these colleges vary in 
enrollment, geographic setting, and type (e.g., public/private, two-year/four-year/university). 


8 See the Institute of Education Sciences’ 2014 Request for Applications, Education Research Grants, 
CFDA Number: 84.305A, p. 43 at https://ies.ed.gov/funding/pdf/2015 84305A.pdf. 
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Exhibit 1-2: College Contexts for PTC Programs Launched by 2018 
PTC Undergrad. 
Program Enrollment 


Location in Fall 2018 


Year 
Launched 


PTC Training Tracks 
Offered 


College Partner 


Type of College 


2010 


2012 


2015 


2015 


2015 


2016 


2016 


2016 


2016 


2016 


2017 


2017 


2017 


Baltimore 
Miami 
Philadelphia 
Seattle 
Jacksonville, 
FL 
Woodbridge, 
VA 

Atlanta 

New York 
Dallas 

Los Angeles 
Phoenix, AZ 
East SF Bay, 
CA 

Chicago 


Boston 


Wilmington, 
DE 


Baltimore City 
Community College 


Miami Dade College 


Pierce College 


Bellevue College 


Florida State College 
at Jacksonville 
Northern Virginia 
Community College 
Atlanta Technical 
College 

Borough of Manhattan 
Community College 
El Centro Community 
College 

West Los Angeles 
College 

Gateway Community 
College 

Diablo Valley College 


Harold Washington 
Community College 
Roxbury Community 
College 

Wilmington University 


Public, 2-year 


Public, 4-year or 
above 


Private not-for-profit, 
4-year or above 
Public, 4-year or 
above 


Public, 4-year or 
above 

Public, 2-year 
Public, 2-year 
Public, 2-year 
Public, 2-year 
Public, 4-year or 
above 

Public, 2-year 
Public, 2-year 
Public, 2-year 


Public, 2-year 


Private not-for-profit, 
4-year or above 


4,523 


54,973 


13,226 


23,413 


50,9292 


3,562 


26,506 


11,729 


11,743 


5,081 


20,271 


8,643 


2,007 


Information technology 
(IT), business operations 


IT, sales, business 
operations 
IT, financial operations 


Business operations, 
software development 
IT, business operations 


IT, business operations 
IT, software development 
Financial and business 
operations 


IT, business operations 


IT, business operations 


IT, business operations, 
software development 
IT 


Financial operations, 
sales 
IT 


Financial operations, 
software development 


Source: Integrated Postsecondary Education Data System (IPEDS) tabulations for institutional characteristics and 


enrollments. 
4 Number represents enrollment at all Northern Virginia Community College campuses. 


Further attesting to the new model’s viability, PTC programs have continued to operate in all 
cities since inception.’ The number of programs and total enrollment have risen steadily since 
the program’s 2010 inception (Exhibit 1-3). By mid-2018, PTC programs were serving around 
2,000 participants per year. Most of the enrollment growth was due to additional programs 


7 In all but one case, programs operated continuously at the same colleges. Year Up relocated the 
earliest PTC program from the Community College of Baltimore County to Baltimore City Community 
College in the program’s second year of operation. 


Abt Associates 


1 Introduction | pg. 7 


To Improve and to Prove 


rather than increased enrollments at operating programs. In 2018, the average enrollment 
across the 15 operating programs was still fairly low: about 130 per site. 


Exhibit 1-3: Number of Young Adults Enrolling in PTC, 2010-2018 (Programs Launched by 
2018) 
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The team visited eight local PTC programs over the course of the study. Evidence from on-site 
interviews and observations supplemented by several rounds of telephone interviews showed 
that all major PTC program components were in place and operating in all locations.® 


Year Up’s approach to screening and developing potential sites explains the PTC’s viability to 
date. A “launch committee” comprised of Year Up national senior staff coordinate site 
development. The process in each location operates on a timetable that is adjusted as 
circumstances warrant up to the point that the committee commits to launch. The process 
unfolds in four stages: 


1. Review statistics and other evidence of the “market potential” of candidate 
locations. Indicators of potential include demographic statistics on the number and 
characteristics of young adults, demand for entry-level workers in Year Up target 


8 As discussed in subsequent chapters, the quality of implementation varies across programs; as of mid- 
2019, no program had achieved high fidelity in implementing all key model elements. 
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occupations, good public transportation access, multiple possibilities for community 
college partnerships, and an active philanthropic community. 


2. Engage in initial discussions with potential local partners and assess the 
“momentum” these interactions generate. The launch committee rates momentum 
with local colleges, employers, and funders as either strong, moderate, or minimal based 
on levels of expressed interest and commitment. 


3. Continue to work with potential partners and gauge when momentum is sufficient 
to proceed to launch. Decisions to proceed typically entail considering the balance of 
signals across conditions. For example, in one site, a strong longstanding corporate 
relationship and good potential for support from national funders convinced the 
committee to proceed despite a relatively weak local philanthropic base and longer likely 
commute times for students. 


4. Proceed to launch. Later stages shift to progressively more operational activity needed 
to launch a site. Twelve months prior to the target launch date, planning staff aim to 
have agreements in principle with local college partners, verbal commitments for 
internships from a good number of employers, an assured source of start-up capital, and 
promising connections to recruitment sources in the community. Six months prior to 
launch, activity moves into high gear on all fronts. The program hires staff, finalizes 
agreements on courses and admissions processes with colleges, commences 
recruitment, and accelerates internship development efforts. 


1.3. Study Approach 


The study team designed this project to address the major objectives of IES’s Development and 
Innovation (D & |) grant program. The grants support projects that develop and document 
evidence-based interventions to improve education outcomes. Closely related IES grant 
programs support effectiveness and replication studies of interventions developed in D & | 
projects. By funding these complementary types of projects, IES encourages and supports 
progression across successively higher levels of evidence-building. 


Many D & | projects start with a proposed intervention clearly in mind. For this project, the study 
team deliberately did not pre-specify a focal intervention. Rather, it designed the project to 
identify focal program improvements as a key early task. This task involved engaging 
practitioners—including Year Up and its college and employer partners—in prioritizing needed 
improvements and then in developing improvements. 


The study team took this approach in part because the intervention of interest was a promising 
multi-component program already operating in multiple locations. Although all basic components 
were in place, the program was not performing at the level intended in a number of important 
respects. This approach to the project enabled testing the idea that closer collaboration with 
practitioners would help us to identify critical improvement needs, mobilize experienced staff to 
generate more effective solutions, and promote a sense of ownership leading to longer-term 
adoption of improved strategies. 
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The team identified three high-priority topics for quick-turnaround improvement studies we 
called “mini-studies”: 


1. Lower-than-desired retention during the first six months of the PTC program (i.e., the 
L&D phase), which Year Up and college informants attributed to academic difficulties in 
college classes. 


2. Uneven quality of internships during the second six months, which PTC program staff 
and participants attributed to varying employer approaches to internships. 


3. Perceived difficulties in promoting both follow-on education and full-time employment 
immediately after program completion—difficulties arising from constraints on 
participants’ time and differences in interests and abilities. 


For each focal issue, the study team used different methods to identify and assess improvement 
strategies in concert with key stakeholders. Chapter 2 describes the process for identifying mini- 
study topics and the methods, data, and findings for each study. The initial years of the project 
were spent largely selecting and conducting the mini-studies. 


In the final two years of the project, the study team shifted its focus to a more global 
assessment of the local PTC programs. This activity included three major tasks: (1) a 
quantitative analysis of implementation fidelity; (2) a small RCT to measure PTC’s education 
and employment impacts in three programs; and (3) a round of in-depth interviews to assess 
perspectives on areas of progress and priority needs for program improvement going forward. 
Chapters 3, 4, and 5 summarize the research methods and findings from each of these tasks, 
respectively. 
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2. Development and Innovation: Three “Mini-Studies” 


A major goal of this project was to apply an evidence-based approach to accelerating 
improvements in Year Up’s PTC program. To this end, the study team engaged a diverse array 
of stakeholders to identify program challenges and apply research strategically to aid Year Up in 
addressing them. The team then conducted focused investigations, or “mini-studies,” of three 
high-priority challenges. 


This chapter first describes the approach to the mini-studies (Section 2.1). The rest of the 
chapter takes up each of the three studies in turn (Sections 2.2 to 2.4), presenting findings, 
recommendations, and resulting program changes by Year Up. 


2.1. General Approach to Mini-Studies 


In developing its PTC program model, Year Up took on the challenge of providing services on 
par with its stand-alone “core” program, but at substantially lower costs that could be covered by 
revenue from employer-sponsored internships. The main adaptations entailed co-locating the 
PTC programs on partner college campuses and relying on the colleges for basic academic 
skills and technical training. Under these partnerships, Year Up has less control over instruction 
and less contact with participants than in its core program. It also faces expectations of the 
college partners that the local PTC programs will help to encourage college persistence after 
the programs. For both the core and PTC programs, meeting needs of its employer partners for 
well-prepared interns is critical for creating enough revenue-generating internships and, if all 
goes well, post-program job offers. 


To identify mini-study topics, the study team conducted interviews and focus groups with a 
diverse array of PTC program stakeholders, including Year Up national and local staff, program 
participants, and college and employer partners. Some discussions were by phone, and others 
occurred during site visits to four local PTC programs. In this outreach, the study team used a 
discussion guide designed to explore the program’s four major phases (recruitment and 
admissions screening, learning and development, internships, and post-program services) and 
important cross-cutting issues. The team also reviewed program documents and statistical 
reports from Year Up’s management information system. 


This outreach generated an initial list of 13 topics (Appendix Exhibit A-1). Through a series of 
discussions with national staff, the team refined the list to three issues that were high priority, 
could be evaluated relatively quickly and well, and were not already the focus of a substantial 
Year Up improvement effort. 


e Mini-Study #1 tackled a central challenge in the PTC program adaptation: establishing 
strong monitoring of academic performance of participants and effectively providing support 
to participants experiencing challenges. 


e Mini-Study #2 assessed strategies for working with employers to develop processes for 
setting up internships that consistently provide high-quality workplace experiences. 
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e Mini-Study #3 sought strategies to reconcile the tensions between Year Up’s traditional 
emphasis on full-time career track employment and the increased emphasis in the PTC 
programs on continuing in college after program graduation. 


For each study, the team worked with Year Up staff to develop quick turnaround evaluation 
strategies that imposed relatively low burden on program staff. They drew on methods from 
multiple evaluation traditions, including randomized controlled trials (RCTs), improvement 
science, design-based implementation research, and continuous quality improvement (Orr, 
1999; Bryk et al., 2011; Russell et al., 2017; Sokovick et al., 2010; Fishman et al., 2013; 
Anderson et al., 1995). 


The study team tailored methods to each mini-study’s focal question, readily available data, and 
resource and time constraints. Mini-Study #1 used n RCT to test staff-developed improvements 
in academic monitoring and supports during the program’s six-month learning and development 
(L&D) phase. Mini-Study #2 drew on in-depth interviews and focus groups with current PTC 
interns to make a qualitative assessment of current strategies and potential improvements for 
setting up successful internships. Mini-Study #3 took a mixed-methods approach—combining 
statistical analysis of college persistence following the PTC and core programs with insights 
from in-depth interviews and focus groups. 


The mini-studies used an iterative approach to generate program improvements. This approach 
included informal reporting to Year Up national staff as findings emerged, and it culminated in 
webinars with Year Up national and local program staff to discuss findings and next steps. To 
support timely reporting, make results easier to absorb, and mesh with Year Up’s accustomed 
communication style, these briefings relied on PowerPoint rather than written reports. Typically, 
the study team sent a detailed slide deck as a “pre-read” about a week prior to each webinar. 
They then prepared a shorter version of the PowerPoint for live presentation and followed up on 
meetings with minutes and notes highlighting emerging recommendations. Year Up national 
staff coordinated the program’s responses to mini-study findings and recommendations. 


In mid-2019, the study team conducted another round of interviews with key national and local 
staff to document program improvements resulting from each mini-study. These final interviews 
captured improvements over a period of one-and-a-half to two-years following the initiation of 
each study. 


As the following sections describe, each mini-study produced promising improvements. 


2.2  Mini-Study #1: Improving Academic Outcomes and Retention 


Year Up’s logic model assumes that strong basic academic and occupational skills are required 
for young adults to succeed in the workforce and continue in postsecondary education. Year 
Up’s PTC program aims to help participants develop these skills through academic and 
occupational courses offered by college partners during a six-month L&D phase of the program. 
Youth who successfully complete L&D apply and further develop these skills during a six-month 
internship in preparation for transitioning to regular career-track jobs. 
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Failure of participants to successfully complete L&D poses three concerns for Year Up. First, it 
commonly signals a failure of participants to master the technical skills needed to succeed in 
their focal career track. Second, participants who fail to complete L&D are not eligible to enter 
internships, which are the primary means through which Year Up earns revenue to cover its 
operating costs.° Third, youth who fail to progress to internships likely will see little or no return 
on their investment of time and effort in the program. 


At the time this project was launched in 2016, most PTC programs were both struggling to meet 
their enrollment targets and falling short of their goals for participant retention through L&D. PTC 
staff frequently found out about participants’ academic struggles and their failing in college 
courses very late in the L&D phase, making it nearly impossible to provide meaningful 
assistance and ensure participants received supports they needed to remain in the program. 


The study team gathered initial suggestions for improvement strategies during outreach to 
prioritize mini-study topics. On selecting this as a mini-study topic, the team worked with three 
local PTC sites to develop and rigorously test promising strategies for monitoring and 
strengthening responses to the young adults’ academic challenges during L&D. As an initial 
step, we worked with local staff to document current practices and identify opportunities for 
improvement. 


Having identified promising strategies, we worked with staff to develop and test effectiveness of 
the strategies in improving participants’ success in college courses, measured by completion of 
L&D and continuation in college post L&D. The study also examined the nature and prevalence 
of challenges participants faced, the strategies staff used to monitor and support academic 
success, and views on the usefulness of various strategies. 


To preview this mini-study’s main finding: 


e The improved academic monitoring and support strategies substantially 
increased retention through L&D and persistence in college immediately after 
L&D. 


Staff credited much of the success of the improvement strategies to the establishment of routine 
processes for sharing information on students’ academic successes and challenges. These 
processes typically involved better information sharing between Year Up coaches and 
participants. This information sharing then allowed coaches to connect struggling participants 
with academic and other supports in a timely manner. An important by-product of the study was 
the creation of a resource toolkit to facilitate coaches’ efforts to monitor and support academic 
performance of participants during L&D. 


2.2.1. Study Design 


This mini-study drew on principles of design-based implementation research (Fishman et al., 
2013), improvement science (Bryk et al., 2015), and social experimentation (Gueron & Rolston, 


8 Under Year Up’s cost model, 83 percent of program participants must advance to the internship phase 
of the program for revenues to cover costs. 
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2013) to design and test strategies for better monitoring and supporting participants in their 
college courses. Design and testing was a three-stage process. 


The first stage entailed developing the improvements to be tested in the study. The study team 
conducted a quick but intensive examination of current practices across the three participating 
PTC programs and reviewed relevant literature. The team then selected three local PTC 
programs that differed in the nature and degree of challenge they faced to develop and 
rigorously test low-cost, high-promise strategies for improving academic success and retention 
during L&D. The strategies intentionally aimed to create more effective monitoring of academic 
performance to achieve quicker identification of participants at risk of failing their courses, which 
would allow timelier and more effective responses. The study team encouraged program staff to 
design improvement strategies for testing that were tailored to local needs and opportunities. 


The second stage of the mini-study entailed two cycles of testing the staff-developed 
improvement strategies. The first cycle tested the strategies that were designed as part of the 
planning stage; the second cycle reflected changes to the initial improvement strategies based 
on experiences during the first testing cycle. During both testing cycles, the team randomly 
assigned half of each cohort of new PTC enrollees to an Improvement Strategies Group that 
would receive the improved monitoring and coaching strategies. The team assigned the 
remaining half to a Usual Strategies Group that would continue to receive existing strategies. 


Over the two testing cycles, a total of 317 participants enrolled in one of the three PTC 
programs participating in the study—154 in the Improvement Strategies Group and 163 in the 
Usual Strategies Group (Appendix Exhibit A-2). Randomization ensured that there were no 
systematic differences in the characteristics of the two groups prior to enrollment (Appendix 
Exhibit A-3). 


The main analyses measured the impact of the improvements on the target outcomes— 
retention through L&D and continuation in college post L&D. The study also gathered additional 
data to support exploratory analyses of impacts on other outcomes and provide context for the 
study findings. These other data sources included bi-weekly monitoring calls with site staff, 
semi-structured observations of coaching sessions, brief surveys of coaches and participants, 
and in-depth interviews with participants and staff. 


To estimate impacts of the improvement strategies, the study team compared outcomes for 
participants in the Improvement Strategies and Usual Strategies Groups, controlling for 
background characteristics at the time the participants applied for the PTC program, site, and 
occupational training track. Because participants were randomized to the two groups, 
differences in average outcomes provide unbiased estimates of the true impacts of the 
improvement strategies tested. 


2.2.2 Background Literature 


Retention is a major challenge faced by two-year colleges throughout the nation. Only about 
one-fifth of students who enroll in a two-year college, and fewer than 20 percent of those 
enrolling part-time, earn a certificate or degree within three years of program entry (Juszkiewicz, 
2017). Among the factors contributing to the low completion rates are poor academic 
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preparation, financial burdens, family responsibilities, housing instability, and food insecurity 
(Sowers & Yamada, 2015; Maroto et al., 2015; Brock & LeBlanc, 2005; Goldrick-Rab, 2010). 


The literature suggests that many of the strategies embedded in the Year Up logic model could 
mitigate some of these challenges. For example, quality academic supports such as tutoring, 
counseling, student centers, advising, and learning communities have been shown to improve 
outcomes (Bishop & Brenneman, 1986; Boyd et al., 1996; Brock & Richburg-Hayes, 2006; 
Gupta, 2017; Scrivener et al., 2009; Weiss et al., 2010; Weissman et al., 2012; Scrivener & 
Weiss, 2009; Mahlberg, 2015). However, the literature does not directly address questions 
about how these various strategies should be implemented to ensure participants’ needs are 
identified and addressed in a timely manner. 


2.2.3 Improvements Tested 


The study team worked with program staff in the study sites to come up with improvement 
strategies targeting three goals: (1) improve access to and tracking of participants’ academic 
performance; (2) use real-time academic performance information to inform coaching of 
participants; and (3) improve participants’ access to needed resources and support services. 


The strategy development process began with guided reflection on current practices and 
outcomes. During facilitated meetings, program staff at each study site brainstormed their 
options for addressing performance gaps. The ultimate product at each of the study sites was a 
set of improvement strategies tailored to local conditions. 


At the outset, the study team and Year Up national leaders deliberately maintained an agnostic 
stance towards the improvements discussed. The approach presumed that local program staff 
working as coaches were in the best position to identify participants experiencing academic 
challenges and help them overcome those challenges.'° 


The PTC staff in the study sites developed the initial plans for implementing the improvement 
strategies to be tested with participants entering the program in July 2017. After one round of 
experience in cycle 1, staff then refined the strategies for cycle 2. Appendix Exhibit A-4 
summarizes the improvement strategies used in cycles 1 and 2. 


Cycle 1. At the outset of the study, staff in all three sites actively encouraged college instructors 
to attend orientation sessions. The aim was to better acquaint instructors with Year Up culture 
and establish communication channels with the PTC staff. Staff also sought to set up 
mechanisms for systematically gathering feedback on participants’ academic performance from 
college instructors.'' Neither of these efforts produced much engagement of instructors with 
Year Up staff. 


In two of study sites, responsibility for gathering this information was assigned to existing staff. 
The third site hired a part-time staff member who was tasked to collect performance data on all 


10 All Year Up staff also serve as coaches. 

11 Due to FERPA restrictions, it was not possible for PTC staff to have direct access to the colleges’ 
learning management systems (LMS). Moreover, not all instructors routinely use their LMS in ways that 
would provide the timely feedback PTC staff were looking for. 
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participants but share information only with coaches of Improvement Strategies Group 
members. 


All three programs included a focus on academics in coaching sessions with participants in the 
Improvement Strategies Group. Coaches also encouraged these participants to emphasize their 
academic work in the final portfolio projects that PTC requires. In contrast, coaching for 
participants in the Usual Strategies Group focused mainly on professional and life skills 
development. Usual coaching practices included a mainly reactive approach when participants 
raised personal, social, and academic concerns. 


Early on, individual coaches working with the Improvement Strategies Group developed their 
own approaches to academic coaching. After observing that coaching practices could be more 
intentional and consistent, Year Up national staff and a member of the study team gathered best 
academic coaching practices from Improvement Strategies Group coaches and distilled them in 
a coaching guide. 


Each study site had modest discretionary funds that coaches could tap to help participants in 
the Improvement Strategies Group who were struggling in their academic courses. One site 
invested in a textbook library to address purchasing delays some participants experienced when 
their financial aid awards were late. Another site initially planned to obtain portable hot spot 
devices to address online access issues faced by some participants, but the site did not follow 
through due to logistical snags. 


Cycle 2. All three study sites refined their strategies for cycle 2. Most significantly, all coaches in 
the Improvement Strategies Group began using a one-page Weekly Academic Coaching Notes 
Sheet. This tool was developed from the prototype academic coaching guide used by coaches 
in cycle 1. It was meant to help coaches facilitate conversations with participants about 
academics. 


Coaches in cycle 2 also had access to The Academic Coaching Binder (Baelen et al., 2020). 
This Binder included tools that coaches working with the Improvement Strategies Group during 
cycle 1 found useful. Some of the tools were adaptations of existing tools, and others were 
created by the coaches working with the Improvement Strategies Group. 


2.2.4 Findings 


The strategies tested proved to be effective in improving retention during L&D. Across all three 
sites, participants in the Improvement Strategies Group had a nearly 10 percentage point higher 
L&D retention rate than did their counterparts in the Usual Strategies Group (79 and 69 percent, 
respectively—see third set of bars in Exhibit 2-1). This impact closed 70 percent of the gap 
between the 69 percent retention rate achieved with the usual approach to monitoring and 
supports and Year Up’s target 83 percent retention rate needed for PTC to break even 
financially. 


Looking separately at each cycle, point estimates appear to be substantially larger for cycle 2, 
although the difference across cycles is not statistically significant (p=.29). The difference 
between the Improvement and Usual Strategies Groups is quite large (14 percentage points) 
and statistically significant (p<.05) for cycle 2. 
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Exhibit 2-1: Mini-Study #1 — Retention of Participants in Improvement Strategies Group versus 
the Usual Strategies Group through the End of the Learning and Development 
Phase by Testing Cycle 


77.6% 80.2% 78.9% 
> 73.3% Beate 69.3%" 


Cycle 1 Cycle 2 Total 


Improvement Strategies w Usual Strategies 


Source: Data on retention from Year Up’s management information system. 


Notes: These data are weighted to account for different probabilities of selection within enrollment blocks. The 
estimates of the mean differences between the Improvement Strategies Groups (treatment groups) are based on 
regression models that included covariates for sample characteristics at the time of program enrollment. Means for 
the Usual Strategies Group are unadjusted. The difference between impacts for cycle 1 and 2 is not statistically 
significant (p-value = .29). Asterisks identify impacts statistically significant in a two-tailed test: *** = 1-percent level; ** 
= 5-percent level; * = 10-percent level. 


Findings for impacts on college enrollment in the month following L&D (Exhibit 2-2) are very 
similar to those for L&D retention. The overall difference between the two groups is 13 
percentage points (67 versus 54 percent) and statistically significant (jp < .05, Exhibit 2-2). 
Again, impacts are larger for participants in cycle 2 than cycle 1, though the between-cycle 
difference is not statistically significant (p=.11). 


Exhibit 2-2: Mini-Study #1 — College Enrollment in the Month Following Scheduled Completion 
of the Learning and Development Phase by Testing Cycle 
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Source: Data on college enrollment from the National Student Clearinghouse. 
Notes: The difference between estimated impacts for cycle 1 and cycle 2 is not statistically significant (p-value = .11). 


In more detailed analyses reported in Appendix Exhibits A-5 and A-6, estimates of impacts on 


L&D completion and college enrollment for the three sites vary but generally are positive. Due to 
small samples, the cross-site differences mostly are not statistically significant. 
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Coaches working with participants in the Improvement Strategies Group during cycle 2 reported 
greater awareness of academic challenges among their coaches than did coaches working with 
the Usual Strategies Group (Maynard et al., 2018). Improvement Strategies Group coaches also 
reported devoting more time in coaching sessions to academic performance and supports and 
less time to other issues than did Usual Strategies Group coaches. 


Participants in the two groups reported taking similar numbers of courses and spending the 
same amount of time on schoolwork per week. Participants in the two groups also reported 
similar views about the level of difficulty of their courses and the adverse effects of working a job 
on academic performance. 


With a few exceptions, participants in both groups reported similar access to PTC program 
supports and similar overall levels of satisfaction with the program. Participants in the 
Improvement Strategies Group rated the quality of the supports from Year Up staff significantly 
higher than did their counterparts in the Usual Strategies Group. 


2.2.5 Initial Recommendations 


In late spring 2018, the study team presented findings and recommendations from Mini-Study 
#1 in a series of webinars to Year Up national and local staff. The webinars encouraged staff to 
react to findings and reflect on experiences and directions ahead. 


e Recommendation 1. The central recommendation was to modify the program’s approach to 
coaching during the L&D phase to include a deliberate focus on academic goals, 
achievements, and challenges. Coaches who worked with the Improvement Strategies 
Group said that they found it very useful to begin each coaching session with a discussion of 
students’ academic progress—what specifically was going well and what was not. They 
found the Weekly Academic Coaching Notes Sheet helpful in engaging participants in these 
conversations. It also provided them a structure for jointly reviewing each participant’s 
academic record in the college’s LMS. They also cited several other tools, which the study 
team ultimately compiled into The Academic Coaching Binder (Baelen et al., 2020), as 
helpful in identifying and addressing challenges. 


e Recommendation 2. A second recommendation was to offer formal staff training on 
academic coaching strategies. The emphasis would be on improving early identification of 
academic challenges and devising timely strategies to help participants address the 
challenges. The training could draw on the myriad resources in The Academic Coaching 
Binder, which includes examples of their application. It also would include professional 
development around team problem-solving on academic challenges and factors that trigger 
or exacerbate performance issues. 


e Recommendation 3 was for program staff to be on the alert for additional ways of 
identifying academic challenges that would complement asking participants directly. These 
might include, for example, designating a staff person to gather and share with coaches 
information from college instructors on participants’ performance; creating a shared 
worksheet containing instructor feedback on performance and academic challenges of 
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specific participants; or encouraging conversations about academic challenges during 
coaching team meetings. 


e Recommendation 4 was for Year Up national staff to consider other applications of the 
evaluation-based improvement process used in this study. This process began with Year Up 
national staff identifying and prioritizing their improvement needs. They then engaged with 
local program staff and the evaluation partners to specify the general parameters of the 
improvement efforts and their evaluation. Next the local program staff, in partnership with 
the study team, took steps to implement the improvement strategies through successive 
rounds of testing and refinement. 


Three features of Mini-Study #1’s design were central to its success. The first is engaging 
stakeholders in selecting improvement goals. The second is working closely with the local 
program staff in designing the evaluation to ensure it produced credible impact estimates with 
minimal disruption of operations. The third is using low-stakes, low-burden approaches to 
monitor the experiences of participants assigned to the Improvement Strategies and Usual 
Strategies Groups. 


2.2.6 Year Up’s Response 


Year Up staff who participated in a May 2018 briefings on Mini-Study #1 reacted favorably to the 
findings and recommendations. Program leads reported that their relatively high engagement in 
the research design fostered ownership in the findings. They were pleased that rigorous 
evidence supported their strategies and hoped to institutionalize the improvement strategies that 
had been effective. Year Up’s national staff planned to share the findings with other PTC 
programs and were upbeat about the potential to learn from rapid turnaround tests of other 
program improvements. 


One year later (May-June 2019), the study team re-interviewed national and local staff leads to 
document any further steps to build on improvement strategies identified in the mini-study. Staff 
in all three study sites reported that they were still using the coaching practices developed for 
the study. They reported that coaching sessions now routinely begin with a discussion of 
participants’ academic progress and challenges. Such discussions often involve staff and 
participants looking together at recent activity recorded in the college LMS. Coaches reported 
that routinized monitoring helps identify problems and their root causes early enough to support 
timely intervention. Participants reportedly appreciate the extra attention from coaches to their 
academic challenges. 


PTC program leads also reported that they had stepped up efforts to solicit feedback from 
college instructors. For example, one site extended its college liaison position to facilitate 
information sharing with instructors. This liaison created a master schedule for the academic 
calendar that helped students and coaches identify strategic time points for checking on 
academic progress. 


Local staff reported that coaches had not expanded their efforts to provide additional tutoring 
through the PTC programs due to the costs and difficulty of providing such services. Instead, 
they stepped up their efforts to connect struggling students with existing college tutoring 
resources. 
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Year Up national staff encouraged staff in other local PTC programs to adopt strategies 
emerging from the mini-study in two online webinars. In the first session, the program manager 
at one of the study sites shared her experiences doing more academically focused coaching 
and the resulting benefits. In the second session, a Year Up national staff member presented 
mini-study findings, previewed a revised version of The Academic Coaching Binder, and 
facilitated a discussion of how other sites could adopt coaching practices like those developed 
and tested in the mini-study. Year Up national staff had not surveyed the extent to which local 
PTC programs made improvements as a result of the webinars, but they cited anecdotal 
evidence of sites later strengthening academic monitoring and supports. 


This first mini-study helped Year Up staff recognize and better appreciate the challenges their 
college partners face in serving students such as Year Up’s co-enrolled participants. It also 
highlighted ways that the PTC partnership can benefit the colleges. For example, community 
colleges typically have far higher student-to-advisor ratios than do the PTC programs. Colleges 
rely heavily on part-time, adjunct instructors, who have more limited opportunities that PTC 
coaches to observe and/or support Year Up students who face academic challenges. The study 
findings suggest that even modest efforts to improve information flows can improve the capacity 
of PTC programs to support participants’ success. 


Finally, both Year Up national and local PTC staff cited aspects of the collaborative 
development and testing of program improvement strategies as rewarding and worth applying to 
other challenges. They liked having their challenges and improvement options framed in a wider 
research context. They believed the emphasis on brainstorming and local expertise led to 
solutions well suited to local situations. And they appreciated the collegial, nonjudgmental 
discussion about the mini-study methods and findings. 


2.3 Mini-Study #2: Setting Up Successful Internships 


Positive internship experiences are critical to Year Up’s success for two reasons. First, for 
participants, well-designed and supervised internships provide valuable opportunities to apply 
technical skills learned in college courses in real work settings, build strong resumes, and meet 
potential employers. Second, successful internships foster repeat business from Year Up’s 
employer partners. This repeat business is important for generating the revenue necessary for 
programs to achieve financial sustainability. 


During the study team’s early outreach, interest in promoting consistently high quality 
internships emerged as a priority issue. There was a sense that employers might welcome more 
guidance on the design and supervision of internships, assuming such guidance could be 
provided without undue burden. 


Year Up national staff and the study team accordingly agreed to focus Mini-Study #2 on 
identifying ways to strengthen support to employers in setting up internships. As background, 
the study team scanned the literature for practices that seemed to be effective in other settings. 
The team visited several PTC sites to understand current processes for working with employers 
to establish internships and how employers were running internships. 


The overarching recommendations from this mini-study were for Year Up to: 
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e Provide employers with more support in developing and delivering high-quality 
internships. 


e Create communities of practice among employers to foster networking, 
brainstorming, and sharing of best practices. 


Such support might include tailored trainings and a living archive of techniques and examples of 
best practices drawn from experience with various Year Up employer partners. This archive 
could be complemented by guidance materials on how to go about setting up strong internships. 
Such materials would cover topics such as onboarding, preparing six-month activity plans, and 
ensuring interns receive constructive feedback regularly and social supports as needed. The 
recommendation also included suggestions for ways Year Up might adjust its internship 
development processes to provide more opportunities for employers to work with program staff 
and other internship providers in designing their internships for PTC participants. 


2.3.1. Study Design 


For this mini-study, the study team drew on best practices cited in the literature and on findings 
from a series of interviews and focus groups with Year Up’s key stakeholders. Specifically, in 
summer 2017, the team visited four local PTC programs to interview staff and employers and (in 
three programs) conduct focus groups with interns. Each focus group included about 10 interns 
and ran about 90 minutes. 


The team drew also from material gathered in previous interviews for this project and a 
companion study of Year Up’s core program (Fein & Hamadyk, 2018).'? This information was 
supplemented with a selection of program data and documents, including information from Year 
Up pre-internship surveys of participants, program guidance documents (e.g., Year Up 
orientation presentations for new internship supervisors, guidance for PTC staff on the process 
for matching interns with employers), and administrative statistics (e.g., number and tenure of 
employer partners, internship completion rates, rates of conversion from internships to full-time 
employment with internship sponsors). 


2.3.2 Background Literature 


A brief review of the literature on workplace learning—covering internships, apprenticeships, 
and new employee training—highlighted several characteristics of successful workplace 
learning experiences. 


One is having clear objectives for the workplace learning. Ideally, internships should prepare 
interns with key requirements and a range of experiences typical of the work a company does 
(Fox, 2014; Van Noy et al., 2016). Employer input in designing objectives is critical for 
developing meaningful simulations of workplace job assignments, which are helpful in preparing 
participants for seamless transitions from classrooms to internships to regular employment 


12 Employer interviews included a mix of internship supervisors and corporate managers responsible for 
the Year Up partnership. Generally, internships are similar for participants in Year Up’s core and PTC 
programs. Some companies host interns from both. 
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(Goldberger, 1994; Rogers-Chapman & Darling-Hammond, 2013). To the extent possible, the 
internship’s learning objectives should align with interns’ career interests and prior coursework. 


A second characteristic of successful workplace learning experiences is that they provide rich 
opportunities to acquire transferable skills needed to succeed in regular employment in the 
target field (Darche et al., 2009; Rogers-Chapman & Darling-Hammond, 2013). This goal is 
likely to be furthered by having college instructors and workplace supervisors collaborate in 
designing skill development activities (Organisation for Economic Development, 2010). 


Third, successful internships often provide opportunities for rotation through different worksite 
placements that offer exposure to varying jobs at the company. Relatedly, it is important to 
minimize downtime during and between rotations to maintain interns’ interest and maximize 
learning opportunities (Gruber et al., 2008; Hamilton & Hamilton, 1997; Organisation for 
Economic Development, 2010). 


Other ingredients in strong internships mentioned in the literature include consistent, 
constructive monitoring and capstone projects that build on skills learned in the classroom and 
onsite (Alfeld et al., 2013; Goldberger, 1994; Organisation for Economic Development, 2010; 
Rogers-Chapman & Darling-Hammond, 2013). 


2.3.3 Current Practices 


Year Up has established practices for identifying employer partners, matching participants to the 
employer partners and internship supervisors, and equipping participants with requisite 
workplace skills. Although these practices generally function well, intern reports suggest that the 
quality of their experiences is inconsistent. At the most general level, most stakeholders 
interviewed said it could be helpful to do more upfront work with employers defining the nature 
and expectations for specific internship positions, particularly the associated learning goals. 


From employers’ perspective, successful internships align interns’ skills with available work 
opportunities and needs of the company. From interns’ perspective, a successful internship is 
one that is interesting, supportive, and good for resumes and job prospects. 


Year Up’s internship development team seeks to secure employer partners that offer internship 
positions (“seats”) aligned with the vocational tracks offered by local PTC programs and the 
technical skills program that participants are expected to have after completing L&D. Internship 
sponsors include a wide range of companies and organizations. Many are large, national 
corporations, but they also include a mix of smaller for-profit and non-profit organizations. 
Employers that have more recently partnered with Year Up tend to have fewer employees and 
sign on for fewer internship seats (often a single seat) than do employers with longer-standing 
relationships with Year Up. 


Employers’ motivations for supporting internships vary, especially regarding the extent to which 
they view their sponsorship as contributing to a talent pipeline. However, they tend to share the 
view that interns should have job skills that allow them to add value to the workplace from day 
one. Some employers noted that they look to interns both for specific contributions to tasks at 
hand and to bring fresh energy and ideas. 
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Before the start of each internship phase, PTC program staff work to match prospective interns 
with employers. They consider qualities of both the intern and sponsoring employer that might 
affect the success of the match. Matching interns’ interests and technical skills with job 
requirements is the paramount consideration. Other factors include the distance from the 
intern’s home to the worksite, the nature and breadth of learning opportunities employers offer, 
and how much support an intern is likely to need and find at various worksites. Program staff 
typically work with incomplete information at both ends (i.e., participant and worksite attributes) 
and optimizing the overall match for a given set of participants and available internship positions 
is reportedly often challenging. 


Most local PTC programs offer some form of orientation for new internship managers. 
Participation is voluntary, and the length and approach to orientation are reportedly quite 
variable. A look at approaches in several sites suggests that orientation sessions typically spend 
the most time introducing employers to Year Up’s mission, describing general attributes of PTC 
program participants, and reviewing administrative processes (e.g., reporting and 
communicating with Year Up). Often there is a brief discussion of what a well-designed and run 
internship entails. 


In interviews with interns and employers, most seemed satisfied with their experiences overall. 
Asked how internships could be improved, some interns said they would have liked to know 
more at the outset about the skills needed to do well in particular internships. Interns also 
suggested a general need for more explicitly planned learning and networking opportunities on- 
site. Based on interns’ and employers’ descriptions of their internships, such opportunities 
seemed to be more common at larger firms with more established internship programs. The 
latter are most likely to have formal structures in place to support interns and their managers. 


One of the most common impediments to quality internships noted by employers and interns is 
a mismatch between interns’ interests and skills and the skills that the internship requires. 
Another common impediment is excessive downtime during internships, when the intern has 
little to do. 


Interns, employers, and PTC program staff offered a variety of constructive responses to such 
issues. Suggestions for minimizing skill mismatch included providing clearer descriptions of 
each position’s requirements and allowing longer lead time to work on skills needed to do well in 
particular internship assignments. Current practices generate a substantial flow of feedback to 
PTC staff about what particular employers are looking for, though recordkeeping practices are 
inconsistent and the quality of this information is typically weaker for newer than for 
longstanding employer partners. 


Staff aim to have all program participants matched to internship positions several weeks before 
L&D ends. In practice, new internship slots often are not secured until close to the target start 
date. In these cases, detailed descriptions of positions are generally not available to aid the 
matching process. Staff recommended building a strong base of long-term employer partners to 
minimize the need to rely on new slots late in the matching process. 
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The team heard many good ideas for how to combat downtime during internships. A general 
suggestion for employers was to work with PTC staff and participants to fashion specific six- 
month activity plans for each intern at the outset. These plans could include provisions for self- 
study/independent learning, for a flexible list of tasks providing additional opportunities as 
needed, and for rotating supervision when an assigned supervisor got too busy. PTC staff also 
could work with participants well in advance to develop plans for alternative growth activities 
during downtime. These plans could include activities such as studying for occupational 
certification exams, homework for online or evening college classes, and meetings with a variety 
of staff at the company to learn more about the varying occupational possibilities there. 


As these examples attest, there is no shortage of promising practices across the gamut of PTC 
employer partners. The problem is that these practices have not been inventoried and made 
widely accessible. Stakeholders agreed that a sharable, living repository of examples and tools 
could be useful to new and longstanding employer partners alike. 


2.3.4 Initial Recommendations 


The mini-study produced two overarching recommendations for improving the consistency and 
overall quality of internships. The first is for Year Up to provide additional support to its employer 
partners in planning, creating, and delivering internships. The cornerstone of this effort could be 
a living archive of guidance and tools on best practices in setting up and managing internships. 
This archive would build on materials already in use across Year Up. 


A second recommendation is to create communities of practice that involve Year Up’s employer 
partners in networking, brainstorming, and sharing. Such communities might involve employers 
within local markets or draw from Year Up’s wider national networks to emphasize particular 
industry or occupational sectors. 


The mini-study also generated four somewhat narrower recommended practices to local staff for 
setting up strong internships: 


1. Create detailed specifications for each internship position well in advance of the 
internship. Such descriptions would identify the needed technical skills, key work 
activities or tasks, skills to be developed, and related employment opportunities at and 
beyond the firm. Year Up could create a template and maintain a library of position 
descriptions. These specifications could help PTC programs and college partners better 
tailor curricula to employers’ needs, improve matching of interns with employers, and 
help participants to prepare for particular internships. More detailed position descriptions 
would help internship supervisors and interns to better plan and manage work 
assignments. 


2. Strengthen training and planning with employers around the kinds of activities 
and supports they would provide—potentially specifying these as part of written 
agreements. The agreements would include provisions to address downtime, social and 
networking interests, and other typical internship challenges. Year Up might create a 
prototype guide and provide resources on developing plans for individual internships. 
Plans would be designed to help interns adjust quickly to the workplace, reduce related 
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anxieties, and maximize the value of the internship experience. Plans also could help to 
ease challenges for supervisors who otherwise might need to spend more time figuring 
out how to help interns acclimate to the firm. 


3. Provide technical assistance to employers and participants in developing 
individualized six-month learning plans for internships. Study informants suggested 
developing six-month plans specifying individualized goals and activities for each intern. 
Year Up could facilitate such a step by providing a template, sample plans, and training 
to interested internship supervisors. 


4. Help employers develop and institute monitoring and feedback protocols for 
internship supervisors and mentors. Worksite supervisors’ monitoring and feedback 
should be aligned with individualized internship plans and include helping interns 
prepare for jobs after they complete the PTC program. PTC staff could offer prototype 
monitoring forms, samples of positive feedback, and coaching in best practices for 
interested supervisors. 


2.3.5 Year Up’s Response 


The study team briefed members of Year Up’s national staff on the above findings and 
recommendations in a March 2018 webinar. Follow-up discussions in summer 2019 explored 
changes over the ensuring year, including responses to study recommendations. 


National staff reported that the recommendations generally had resonated with staff and 
seemed actionable. Staff saw good potential for leveraging existing infrastructure and processes 
to support the proposed improvements. Their main concern was to ensure that doing so did not 
add too much burden for employers or local PTC staff. 


Year Up’s main focus in this arena in 2018-2019 was solidifying working relationships with new 
and longstanding employer partners. Related efforts focused on improving existing processes 
critical to the basic functioning of employer partnerships, processes related to securing 
internship positions and payments, matching students to positions, familiarizing employers with 
Year Up, and setting up monitoring protocols (e.g., attendance reporting and periodic online 
assessments by supervisors). 


National staff noted several steps to improve internship setup and monitoring along the lines 
recommended. For example, Year Up created a repository of templates describing the work that 
typical internships involve. The goal is to facilitate communication with employers and 
participants about how particular internship experiences and future job opportunities might align. 
It also developed training videos for new internship managers that, among other things, identify 
more and less productive practices in running internships. '° 


Staff also described ongoing efforts to improve orientation of new internship managers. Year Up 
national staff created a slide deck and accompanying facilitator’s guide for highlighting the 
ingredients for successful internships. The new training were piloted with two large employer 


13 See materials at https://gradsoflife.org/opportunity-manager-training/. 
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partners in Year Up’s Dallas market with the goal of rolling out the training to all markets with 
the following cohort of interns.'* At the time of the conversations, Year Up national staff were 
creating a new web portal to allow internship supervisors to connect with one another within and 
across employer partner sites. 


As noted above, Year Up’s main priority has been to strengthen basic mechanics of the 
internship relationship and processes. In the longer term, leads anticipate revising templates for 
employer agreements to encourage greater specificity about the objectives and skills required 
for each internship position. They also would like to move ahead with templates for 
individualized six-month internship plans, as recommended above. 


2.4 Mini-Study #3: Fostering College Persistence 


In the short term, Year Up aims to equip participants in all of its programs with enough training 
and work experience to secure entry-level jobs in growing fields following program graduation. 
Strong relationships with employers are critical to the program’s theory of change and financing. 
The organization also recognizes that ongoing skill development is crucial for longer-term career 
advancement. For these reasons, Year Up also seeks to promote college persistence among its 
program graduates. 


In the Year Up core program, students received college credit for Year Up training through 
cooperative agreements with local college partners.'® But the core program’s main emphasis, 
reflected in post-program performance goals, is on full-time employment, not college 
persistence. Consistent with this emphasis, impact findings for Year Up’s core program show 
earnings gains for participants but no increase in their college enrollment rates after the one- 
year program (Fein & Hamadyk, 2018). 


The PTC program puts greater emphasis on college persistence than does the core program. 
The local PTC programs operate in college settings where the primary orientation is towards 
classroom learning, credit accumulation, and credential completion. Indeed, PTC’s potential 
contributions to increased college enrollment, persistence, and completion explain a good deal 
of college partners’ interest in the model. 


The key question for this mini-study was whether and how the PTC programs could do more to 
improve college persistence and degree attainment among program graduates without 
compromising post-program employment outcomes. Impact findings for the core program 
suggest that it can be challenging to foster increases in both work and school simultaneously 
(Fein & Hamadyk, 2018). Year Up’s national staff continue to see successful transition to 
career-track employment as the paramount goal for PTC participants and do not want to 
encourage college persistence at its expense. This third mini-study sought to identify ways the 
PTC programs can address college partners’ interest in having PTC participants continue in 


14 The website is at https://yusupervisorhub.com/. 

15 Year Up recently revised its approach to securing credit for core program courses. Students now earn 
college credit through the American Council on Education’s Credit Recommendation Service, rather 
through agreements with local colleges. 
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college after completing their internships without compromising on Year Up’s priority on post- 
program employment. 


The central recommendation from this mini-study was for Year Up to: 


e Introduce a systematic approach to career planning that would balance and 
further both the college persistence and career-track employment goals. 


The recommendations envision local PTC programs working with participants to develop 
personalized career plans including specific steps for both employment and continuing 
education. Ideally, these plans would be developed in concert with PTC program staff, college 
advisors, and worksite supervisors and, subsequently, reviewed and revised periodically as 
participants gain more experience and exposure to work environments. The plans would identify 
sequences and combinations of work and school experiences that are consistent with each 
participant’s goals. Progress of participants would be monitored against their individual plans, 
rather than common expectations regarding goals for work and school. Year Up would use the 
plans to monitor local programs against specified targets for percentages of graduates who are 
on track with their individual career plans and engaged at least full-time in plan-specified 
activities. 


2.4.1. Study Design 


This mini-study drew on several types of information, including a brief literature review, analyses 
of Year Up administrative data, and interviews and focus groups with PTC program 
stakeholders. Statistical analyses also used college records from the National Student 
Clearinghouse (NSC). The analyses determined rates of college persistence by former Year Up 
program graduates and explored factors associated with persistence. '® The analysis focused 
primarily on former participants in PTC programs but also included comparisons with former 
participants in Year Up’s core programs. 


The study relied primarily on NSC data on college enrollment for the 1,029 PTC participants 
who enrolled prior to July 2015, including 667 graduates with four or more months of post- 
program NSC follow-up data. Of this sample, 529 participants enrolled prior to July 2014 and 
thus had 18 months of follow-up data. To put PTC’s greater emphasis on college persistence in 
perspective, the study also used comparable NSC data for 7,288 young adults who enrolled in 
Year Up’s core program prior to July 2015 (5,418 of whom graduated). 


16 This work also examined the quality of data on college enrollments captured at Year Up’s four-month 
post-graduation follow-up, including comparing program data with college records in the NSC. 
Examination of data from staff reports and NSC records showed that the program data tended to 
overstate and the college data to understate post-PTC enrollment levels. Of the two sources, the team 
judged the NSC’s data collection processes the more reliable. NSC’s chief source of bias—failure to 
match some participants to NSC records due to discrepancies with Year Up IDs—might be addressed 
to asmall extent by repeated matching on Social Security number and then name and date of birth. 
Since NSC included college records during the program for only 89 percent of a sample of PTC 
graduates (all of whom were enrolled in theory during the period), 11 percent may be a reasonable 
estimate of under-coverage. Thus, multiplying observed enrollment rates by 1.12 (100/89) provides a 
rough correction for false non-matches. 
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The mini-study also draws on interviews and focus groups conducted in 2018 that explored 
approaches to college persistence and other subjects. The study team visited three local PTC 
programs to interview program and college staff and conduct focus groups (each involving 
around 10 participants currently in internships). This study also draws on interviews with 
employers conducted for the overall PTC study (six interviews) and for the parallel PACE study 
of Year Up’s core program (12 interviews). 


2.4.2 Background Literature 


The literature on returns to college credentials reports that, increasingly, a college degree is 
required to obtain professional jobs providing good pay and benefits. Although economic returns 
are largest for bachelor’s and graduate degrees, there also are appreciable returns for 
associate’s degrees and certificates—particularly in technical fields (Belfield & Bailey, 2017; 
Binder & Bound, 2019; Carnevale et al., 2010, 2013; Dadgar & Weiss, 2012; Fuller et al., 2017; 
National Research Council, 2014). 


Unfortunately, degree and certificate completion rates are low at public two-year community 
colleges, which are most accessible for low-income students. Only half of first-time, degree- 
seeking students at such schools have earned a credential or are still enrolled after six years 
(National Student Clearinghouse Research Center, 2019). One reason is that low-income youth 
are more likely than others to attend college part-time while working full-time. Moreover, those 
who enroll in college part-time are less likely to earn a credential, and for those who do earn a 
credential, the average time-to-degree is longer (Attewell & Monaghan, 2016; Bound et al., 
2012; Jepsen et al., 2010; National Research Council, 2014; Stinebrickner & Stinebrickner, 
2003). 


In response, some colleges have worked with researchers and non-profit organizations to test 
approaches for supporting full-time college enrollment—for example, through intrusive 
guidance, additional financial assistance, and other supports. Rigorous evaluations have found 
that such programs can increase college persistence and completion (Gupta, 2017; Roder & 
Elliott, 2019; Rolston et al., 2017). To date only one of these studies (Roder & Elliott, 2019) 
measured impacts on earnings. That program (Project QUEST) produced substantial earnings 
gains persisting at least nine years. 


The literature does not provide much guidance on how to support longer-term skill acquisition 
when full-time college enrollment is not possible—as is the case for most of the disadvantaged 
young adults Year Up targets, who must work full-time to make ends meet. The options include 
Ongoing supports for part-time enrollment in existing college programs and alternative forms of 
training that may be a better complement to work. The latter include progressive increments of 
accelerated training (similar to what Year Up provides to participants during the first six months), 
apprenticeships, and other work-based approaches. 


The evidence on these alternative approaches is limited. Findings on Year Up and other 
intensive, occupation-specific programs (often referred to as “sectoral” training approaches) are 
encouraging. But few studies of these programs have examined the persistence and magnitude 
of benefits over the long term. And there has been little testing of approaches that entail multiple 
increments of intensive training separated by spells of full-time work. 
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2.4.3 Descriptive Analyses of College Persistence 


On average, 62 percent of PTC program participants enrolled by July 2015 graduated from the 
program, and about half of them continued in college after graduating the program.'’ Examining 
the subset of participants who graduated from the earliest PTC enrollment cohorts (N = 357) 
affords a look at persistence over a longer time horizon. Of them, most who were enrolled four 
months after finishing PTC were still enrolled a year later (Exhibit 2-3, PTC panel, red line). 


Exhibit 2-3: Mini-Study #3 — Percentage of Participants Enrolled in College During the Nine 
Quarters after Enrolling in Year Up’s PTC and Core Programs, by Program 
Completion Status 
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Source: Year Up administrative data and data from the National Student Clearinghouse. 


Notes: The sample includes program enrollees through July 2015. Quarter 0 is the enrollment quarter. The actual 
date of enrollment within the calendar quarter varies across programs and over time within programs. The learning 
and development phase of the program corresponds to the six months after enrollment; the internship phase 
corresponds to the second six months after enrollment; the post-program period begins sometime during quarter 8, 
depending on the program enrollment date. 


Young adults from the same PTC cohorts who did not finish the program were about 20 
percentage points less likely to remain in college (blue line). The reasons the latter left college 
are likely to be closely linked to factors that often impede success in PTC generally, such as the 
academic challenges addressed in Mini-Study #1 and other personal and family issues. 


Year Up’s core program encourages graduates to enroll at local partner colleges. However, 
compared with participants in the PTC programs, college enrollment rates among participants in 
local core programs are substantially lower during and after the program for both graduates 
(Exhibit 2-3, Core panel, red line) and dropouts (blue line). 


17 College persistence estimates are based on NSC college enrollment records. Direct estimates show 
that of 667 PTC graduates, 45 percent were enrolled in college four months after graduating. A more 
accurate estimate, adjusting for false non-matches (see footnote 16), is 51 percent. 
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The team also analyzed the characteristics of PTC participants that are related to post-program 
college persistence. Employment status as of Year Up’s four-month post-program follow-up was 
one of the strongest predictors. As shown in Exhibit 2-4, PTC program graduates who were 
working part-time immediately after the program were more likely to remain enrolled in college 
than those who were either working full-time or not working. Several other subgroups of PTC 
graduates also had higher college persistence rates (not shown in exhibits)—notably those in 
the youngest age group (<20) and those with some college experience before entering PTC. 


Exhibit 2-4: Mini-Study #3 — Percentage of PTC Program Graduates Enrolled in College During 
Each of the First Nine Quarters After Enrolling in PTC, by Employment Status in the 
Quarter Following Graduation 
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Source: Year Up administrative data, data from the National Student Clearinghouse, and data from the National 
Directory of New Hires. 


Note: The sample includes all program participants enrolled between January 2011 and July 2015. Quarter 0 is the 
enrollment quarter. The actual date of enrollment within the calendar quarter varies across programs and over time 
within programs. The learning and development phase of the program corresponds to the six months after enrollment 
(roughly QO-Q1); the internship phase corresponds to the second six months after enrollment (roughly Q2-Q3); and 
the post-program period begins sometime during Q3-Q4, depending on the program enrollment date. 


PTC graduates whose internship sponsors offered them jobs also were more likely to continue 
in college than were their counterparts who did not get a job with their internship sponsor. It is 
unclear why persistence should be higher for young adults whose internships directly led to job 
offers. One possibility is that they have relatively high career aspirations and fewer life 
challenges, rendering them better able to combine work and school. Such participants also may 
have more time to focus on school given less need for time-intensive job hunting after 
completing PTC. Employers who hire interns may be more able and willing than other 
employers to support continuing education of their employees. 
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College persistence varied substantially across local PTC programs, even after controlling for 
participant characteristics. This variation raises the possibility that the philosophies and 
practices of college partners and Year Up programs may influence college persistence rates. 
For example, views of the value of continuing in college after leaving the PTC program may vary 
across programs, and college partners may differ in the degree to which they actively support 
ongoing college enrollment among PTC graduates. 


2.4.4 Stakeholder Perspectives 


As fieldwork for this mini-study evolved, it became clear that, in order to understand post- 
program outcomes, it was necessary to explore how college goals and supports were portrayed 
to participants. Thus, the interviews and focus groups inquired about both goals and supports 
for college enrollment from recruitment through internships and into the post-program period. 
We explored stakeholders’ views on how each phase of the PTC program operated and how it 
might be possible to strengthen expectations and supports for continued enrollment in college 
through internships and beyond. 


During recruitment, many PTC programs focused their marketing efforts on the general 
community. Although there were some instances of joint recruitment with the college partners, 
college partners generally did not play a substantial role in program outreach and recruitment. 
PTC program and college staff cited Year Up’s expertise in engaging underserved young adults 
and limited college recruitment resources as main reasons for leaving recruitment primarily to 
Year Up. 


In focus groups, PTC participants reported hearing and responding to pitches for the program in 
varied ways. Although nearly all aspired to earn college credentials eventually (typically at least 
an Associate’s degree), for some, the main appeal of the PTC program was its promise of a 
well-paying professional job after graduation. Other PTC participants—including many who had 
previously struggled in college—said that the program’s offer of a supportive start towards a 
longer-term college certificate or degree was the most persuasive factor. 


Most PTC participants in the focus groups said they did not have well-defined career plans 
when they entered the program. Some said that initially they also did not have a strong interest 
in Year Up’s occupational tracks. In interviews, program staff reported that weak or uncertain 
career interests often made it difficult to assign enrollees to suitable PTC occupation tracks and 
college courses, and later to internship positions. 


The L&D phase of the program offers important opportunities for participants to shape their 
career plans, acquire knowledge and skills needed for college success, and earn credits 
towards a longer-term college credential. In addition, success in college courses during the PTC 
program is likely to engender skills, motivation, and college credit supporting continuation in 
college after graduation. 


The mini-study identified a variety of challenges in supporting college success during the PTC 
program. One theme cited by some program staff and participants was a need to improve the fit 
between participants’ interests, the occupational tracks to which they were assigned, and the 
college courses they enrolled in. Both program staff and participants believed that tightening 
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these connections would increase motivation and persistence during the program and generate 
more credits applicable to desired longer-term credentials. 


A second theme, explored in depth in Mini-Study #1, is the importance of coordination between 
PTC program staff and college instructors to identify and address students’ academic difficulties 
in a timely manner. On the curriculum front, PTC program staff believed that mechanisms to 
ensure that feedback from employers was used to improve college curricula and instruction 
could be strengthened. 


A third theme was a sense of missed opportunities for teaming with college advisors and 
worksite supervisors to foster career planning, including longer-term education plans. Program 
participants reported that college advising services mostly followed the traditional model of 
infrequent, student-initiated sessions focused on near-term course selection decisions. There 
was reportedly some, but not extensive, engagement in career planning with college advisors 
assigned to PTC participants. Communication between program staff and employers focused 
primarily on securing internship slots and troubleshooting difficulties during internships, with little 
discussion about interns’ interests and career plans. 


Program staff mentioned other underexploited opportunities to support college persistence 
during the internship phase. For example, staff routinely encouraged participants to take college 
courses (night or online courses) during internships, but programming that might help structure 
course planning and support completion was mostly undeveloped. Similarly, Wednesday 
afternoon workshops back at the PTC program office during the internship phase spent some 
time on career exploration and goal-setting. Such sessions reportedly concentrated on 
employment—uwith limited, if any, attention to next steps in college and with minimal 
participation by college advisors. There were exceptions. For example, at one site, the college’s 
designated advisor took active steps to encourage interns to continue at her college. 


In focus groups, interns said it could be hard to find a good career coach. They found that, 
although college advisors were knowledgeable about follow-on programs at their colleges, they 
did not Know so much about employment opportunities and pathways in particular occupations. 
PTC program staff had strong knowledge of Year Up’s target occupations, but they had limited 
command of the requirements and steps for obtaining the college credentials needed to 
advance in them. 


Participants in one focus group reported they had received mixed messages from their 
internship supervisors about continuing in school. Some interns said that their supervisors 
strongly encouraged them to finish college after the program. Others said supervisors talked 
about co-workers who achieved success through experience and industry certifications, which 
left interns uncertain of the value of continuing in college. 


Regardless of influences before and during internships, most PTC participants in focus groups 
aspired to earn a college degree. About half planned to attend college part-time while working 
full-time after they graduated from the program. The rest were roughly divided between those 

planning full-time school and those who whose plans were uncertain. 
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Some employers reported that they regularly encouraged interns to finish college after PTC. 
Several cited company-provided tuition and other supports that had helped former PTC program 
graduates earn degrees while working. Other employers said that because their companies 
required degrees for entry-level jobs, they encouraged program graduates to first finish their 
degrees and then apply for employment. 


Year Up provides program participants with up to a year of employment-focused monitoring and 
supports following graduation. Historically, these services have not proactively supported 
college persistence. For most of the period covered by this research project (2015 to mid-2019), 
Year Up encouraged local offices to meet a 75-percent college persistence target for PTC 
program graduates. Local office leads were aware of the target but characterized their efforts to 
meet it as limited. In July 2019, national leaders dropped this performance standard in favor of a 
planning approach emphasizing more individualized education and training goals. 


2.4.5 Initial Recommendations 


The study team discussed its recommendations for increasing college persistence with Year Up 
national staff in a November 2017 briefing. The central recommendation was to undertake 
improvements such that every young adult completing the PTC program would have a clear and 
actionable career plan. The plan should be developed with substantial involvement of college 
advisors and, where appropriate, internship supervisors. Each plan should specify next steps for 
both education and employment, with steps depending on the participant’s preferences, 
abilities, and selected career path. 


Further, because steps would vary, program performance monitoring should not specify uniform 
levels of employment or education. Rather, the recommendations suggested that measures of 
career progress focus on the percentages of graduates who were “on track” with their career 
plans and engaged full-time in work or education or both consistent with the plans. 


The recommendations suggested roles for Year Up and its college and employer partners in 
collaborating on a “guided career pathways” approach that identifies both the educational and 
employment steps critical to pursuing desired careers. Such an approach involves identifying 
and mapping each participants’ career interests to one of a limited set of “pathways to success” 
at each site. Each pathway identifies a specific bundle of college courses and aligned internship 
positions. Each pathway also targets particular jobs and career ladders in specific occupations. 


The study team recommended reworking templates for memoranda of understanding (MOUs) 
between Year Up and its college partners to specify joint recruitment activities that include 
accessing college applicant pipelines. The MOUs and accompanying materials would articulate 
the benefits of PTC program enrollment for existing college students (e.g., improved retention, 
completion, and employment success) as well as the PTC program’s expected contributions to 
recruiting new students. The revised MOU template also would specify expectations for 
communication and coordination between the PTC program and college staff in developing and 
monitoring career plans, participating in the PTC program learning communities, and advising 
participants during and after the program. 
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The study team identified a series of steps to make career planning a more central, ongoing 
process throughout the PTC program experience. The most explicit activities would occur in the 
Career Development and Internship Readiness course during L&D and in internship workshops. 
These activities would engage participants, PTC program staff, college advisors, and worksite 
supervisors in career planning. Upon graduation, each program participant would have a 
detailed career plan and a connection with a dedicated career coach—ideally a college advisor 
or worksite supervisor—to support follow-through. Year Up staff would continue to actively 
monitor each participant’s progress, update plans as needed, and ensure that the participant 
has an engaged career coach. 


The study team encouraged Year Up to pilot more intensive versions of a guided career 
pathways model in selected programs throughout its network. In addition to strengthened 
activities during the program, such pilots could test expanded supports for education and 
training after a participant graduated. For example, employers might be encouraged to extend 
existing supports for education and training through simple policy changes such as eliminating 
the waiting period before new hires qualify for tuition support. Pilots also might experiment with 
supports that encourage employers to maintain relationships with former interns they do not hire 
upon graduation, but who are working towards college degrees. External funding for pilots could 
be used to cover these enhanced services and related evaluation costs. 


2.4.6 Year Up’s Response 


At the November 2017 webinar, meeting participants responded favorably to the central 
recommendation. National staff also liked the idea of making “being on track with career plans” 
a key performance benchmark, rather than Year Up’s current practice of gauging program 
performance against a common post-program college enrollment rate target. However, they 
were wary of revising performance requirements pertaining to employment, as the employment 
emphasis is central to Year Up’s mission and theory of change. 


The study team interviewed key national staff in May to June 2019 to ascertain what changes 
Year Up had made in response to the Mini-Study #3’s recommendations. Informants reported 
substantial progress in implementing the study’s central recommendation—to develop and 

implement an overarching career planning approach that ensures each PTC participant has a 
clear and actionable plan for next education and employment steps upon completing Year Up. 


At the center of the new approach—now operating in all PTC and core programs—is a new 
short course in career planning taught over two weeks during the Wednesday Internship 
Seminar. These classes are highly interactive and activity based. Participants start with group 
activities focused on planning for hypothetical young adults and then complete a series of 
worksheets and essays that culminate in their own plans. To prepare their plans, participants 
research occupations of interest, interview professionals in these occupations, and reflect on 
and discuss goals with Year Up coaches and classmates. 


The cornerstone of each plan is a worksheet in which the participant identifies short-term, mid- 
term, and longer-term goals for employment, education and training, exam-based certifications, 
and other important areas of life. Participants enter plan highlights in Year Up’s Salesforce® 
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database, establishing the basis for future monitoring of their career progress (discussed 
below). 


Instructors and PTC program coaches encourage participants to include meaningful education 
and training goals. Participants decide whether and what their goals should be. Year Up has 
developed materials to train its own staff coaches in how to guide participants in developing 
their plans. Local PTC programs have not yet made concerted efforts to engage college 
advisors or worksite supervisors and mentors in plan development, as was recommended. 


A new program performance goal stipulates that 100 percent of participants should have 
completed plans entered into Salesforce® as of graduation. Soon a related performance goal 
will specify a standard for the proportion of a program’s participants who are “on track” with their 
plans 12 months later. 


Year Up had neither determined how this second outcome will be measured nor specified the 
target achievement rate as of our mid-2019 interviews. The organization will continue to monitor 
education enrollment rates post-graduation. As mentioned earlier, though, it no longer will hold 
local offices to the 75 percent college enrollment standard. 


This guided pathways approach to career planning raises important questions for future 
research: 


e How much, and what kinds of, education will participants commit to in their plans? Without 
explicit standards for levels and intensities of planned education, will many participants opt 
to pass on more effortful forms of education and training? 


e What level of guidance and support will participants receive? Year Up has taken important 
steps to train its staff to help participants develop, monitor, and update their plans. It is 
important to assess the quality and results of the practice. 


e To date, there have been only limited efforts to engage college advisors and worksite 
supervisors as potential career coaches for program graduates. Efforts to expand employer 
supports for ongoing education post-program also have been limited, though national staff 
are working on a pilot involving front-ending tuition remission programs offered by several 
employers. Research on innovative approaches to such collaborations could be useful. 


A guided career pathway model ideally would begin career planning during program recruitment 
and enrollment and strengthen alignment between Year Up’s occupational tracks, specific 
internship positions, and follow-on education and training programs. National staff cited several 
efforts under way to strengthen such alignment. For example, efforts in launching newer sites 
have involved more extensive work with college partners to create certificates for completing 
PTC that directly connect to specified Associate’s degrees. In addition to longer-term benefits 
for participants, such credentials are helpful to colleges because they count towards the 
colleges’ overall completion rates. 
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3. Assessing Implementation Fidelity 


This chapter assesses the degree to which the Professional Training Corps (PTC) program was 
operating as intended by the end of the four-year study period. As seen in earlier chapters, 
implementing PTC required Year Up to adapt an intensive, multi-faceted stand-alone program to 
operate in college settings. The aim was for PTC to meet the same participant outcome goals 


set for Year Up’s “core” program but at a substantially lower cost. 


Year Up routinely measures many aspects of its local programs’ performance against specified 
quantitative targets. For other dimensions of performance, standards for successful 
implementation are strongly implied by the program design but neither defined nor routinely 
measured. This chapter covers both types of requirements. For the latter, we offer a preliminary 
formulation of what high fidelity entails and, where possible, provide pertinent statistics. 


The study focuses on data through 2018 to early 2019. Where available, we provide 
comparable statistics for Year Up’s core program. Such comparisons help to see where PTC is 
in relation to its goal of matching the core program’s performance. The findings reported in this 
chapter are based on two data sources—administrative statistics from Year Up’s management 
information system and an online survey completed by all 15 PTC local program leads and the 
college partner liaisons for these programs. The survey—fielded in mid-2019—included 
questions designed to assess aspects of implementation operations for which data were 
otherwise unavailable. 


To preview the chapter’s main findings, by mid-2019 PTC had achieved high fidelity on a 
number of key standards while performance gaps remained for others. Areas of high 
performance include lowering per-participant costs relative to those for the core programs, 
operating with a lean staff, recruiting youth fitting the target profile, collecting full revenue for 
internships, and having high proportions of graduates in either well-paying jobs or school. 
Recruitment and retention were the main areas where performance gaps remained. 


The study examined four broad sets of fidelity indicators. The first set concerns several cross- 
cutting implementation requirements: financial operations, staffing, and college partnerships 
(Section 3.1). The second set pertains to recruitment and admissions (Section 3.2). The third 
set involves implementation of key services during the L&D and internship phases (Section 3.3). 
The fourth set concerns post-program outcomes (Section 3.4). The chapter concludes with a 
brief summary of findings (Section 3.5). 


3.1. Cross-Cutting Dimensions 


High-fidelity operation of the PTC model requires strong performance on three dimensions that 
cut across all aspects of the program. First, local PTC programs need to break even 
financially, which means operating at costs that can be covered by internship revenue from 
employer partners—a level that is about half the operating costs of Year Up’s core programs. 
Second, programs need to operate as well as the core program with a leaner staff, which 
requires that they delegate some and share other responsibilities with their college partners. 
Third, programs need to establish high-quality college partnerships that can create and 
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sustain high-quality admissions processes, develop and deliver strong curricula and instruction, 
and secure suitable office space. 


3.1.1. Reaching Financial Break-Even 


At high fidelity, local PTC programs would run at an average cost per participant that did not 
exceed the average revenue received from employer internship payments. Year Up’s financial 
model specifies the cost and revenue figures that programs need to achieve break-even. 


On the cost side, programs are expected to reduce spending to no more than $20,000 per 
participant once they grow their program enrollment to 160 participants or more per year. 
Spending projections start somewhat higher than this when programs begin, but per-participant 
costs are expected to fall as program enrollment grows. On the revenue side, Year Up’s 
financial model assumes that, in order to generate $20,000 per participant from internship 
revenue, the programs will retain at least 83 percent of participants through the L&D phase and 
place them in internships. It also assumes that employer sponsors pay the full sponsorship rate 
of $26,000 per intern for 90 percent of internships. 


In fiscal year 2018, the average per participant cost was $22,247 across all 15 PTC programs 
and $19,388 for the four programs with enrollments at or above the threshold of 160 
participants. For comparison, the average cost per core program participant was $34,534 in the 
same year. Both the absolute and comparative costs for the PTC programs indicate that, 
overall, the programs achieved high fidelity to the PTC financial model. 


Average revenue was $14,049 per participant for all PTC programs and $13,433 for programs 
serving 160 or participants a year. These amounts cover 63 and 69 percent of costs, 
respectively—substantially less than the 100 percent goal. The chief reason is that the PTC 
programs did not retain a sufficient fraction of participants to the revenue-generating internship 
phase of the program (see further discussion in Sections 3.3 and 3.4). In comparison, the core 
programs achieved higher retention and generated $19,944 in average revenue per 
participant—covering 58 percent of its average per participant cost of $35,434.'8 


3.1.2 Operating Well With a Leaner Staff 


A major source of anticipated lower per-participant costs of the PTC programs compared with 
the core programs is the former’s lower staff sizes. According to Year Up staff rosters, as of the 
end of 2018, the average staff size was 15.5 across the 15 PTC programs (Exhibit 3-1, middle 
panel).'? This figure is very close to the 16.3 average implied by Year Up standards, given the 
number of learning communities PTC programs were serving in 2018. 


18 This cost per student for the core program represents an increase over the $28,290 reported for 2013- 
2014 in PACE. Nonetheless, revenue from employers covered virtually the same portion of costs (58 
percent, compared to 59 percent in PACE), indicating that Year Up was able to increase internship 
revenue at a rate that costs increased. 


18 In 2018, standards were 13, 19, and 26 full-time staff equivalents for stand-alone PTC offices with one, 
two, and three learning communities, respectively. (The standard learning community size is 40 
participants.) Standards for PTC differed slightly in so-called “dual channel” cities — that is, cities where 
Year Up operated both PTC and core programs. 
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Exhibit 3-1: Fidelity of the PTC Programs on Cross-Cutting Dimensions of Implementation 
Performance Level 


PTC OF 0) (:) 
High Fidelity Standard WI EERSTULC) Programs Programs 
Financial Goals 
Costs substantially below core 


program Average per-participant cost FY 2018 ($) 
All offices 22,247 34,534 
PTC programs with 2+ learning communities 19,388 NA 
Cover cost through internship 
revenue Internship revenue per participant in FY 2018 ($) 
All offices 
Amount ($) 14,049 19,944 
% of average total cost 63.2 57.8 
PTC programs w/2+ learning communities 
Amount ($) 13,433 NA 
% of average total cost 69.3 NA 
Staffing Goals 
Staffing close to Year Up standard Average number of staff per office as of December 2018 15.5 NA 
(16.3 staff per office at 2018 
enrollment) 
Local PTC leads deem staffing Availability of staff type cited as creating challenges 
sufficient never/only occasionally (%) 
Local leadership 66.7 NA 
Recruitment/admissions 66.6 NA 
Program operations 80.0 NA 
Employer-focused 93.3 NA 


PTC-College Partnership Goals 
Leaders assess partnership as high 


quality PTC program leads’ assessment of partnership quality 

(%) 
Above average 16.7 NA 
Average 54.4 NA 
Below average 28.9 NA 

College leads’ assessment of partnership quality (%) 
Above average 42.9 NA 
Average 40.5 NA 
Below average 16.7 NA 


Sources: Administrative statistics provided by Year Up, mid-2019 survey of PTC program leads and college liaisons. 


The current standards represent an increase in staffing levels since the beginning of the study 
period, largely in the admissions and student services categories.®° Whether current levels will 
be sufficient to address persisting recruitment and retention challenges is unclear. Findings from 
Mini-Study #1 (see Chapter 2) provide grounds for optimism. That test of staff-designed 
strategies for improving academic monitoring and supports showed that it is possible to 
substantially boost L&D retention without adding staff. 


20 See Chapter 5 for discussion of recruitment challenges facing admissions staff. Year Up initially 
planned to refer participants with social service needs to college services, but found that college 
offerings often were not sufficient. 
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In a spring 2019 survey administered by the study team, PTC program leads reported that 
staffing levels were generally sufficient. However, judgments varied on needs for particular 
types of staff. For example, program leads were least likely to report reported that they had 
sufficient leadership and recruitment/admissions staff (67 percent in both instances), more likely 
to report that general operations staffing was sufficient (80 percent), and highly likely to report 
that employer-focused staffing was sufficient (93 percent). 


3.1.3. Forging High-Quality College Partnerships 


This study did not collect objective measures of college partnership functioning. The spring 
2019 survey of PTC program leads and local college counterparts did ask respondents to rate 
the quality of their Year Up—college partnership, however. 


The survey asked PTC program leads to compare their site to other PTC program sites, and it 
asked college liaisons to compare their PTC partnership with other partnerships they have with 
other external organizations. Relatively few PTC program leads (17 percent) rated their college 
partnership as above average, and most (54 percent) rated their site as average (Exhibit 3-1 
above). In comparison, 43 percent of college leads rated their partnership with the PTC program 
as above average, and 41 percent rated it average. 


These numbers hint at lower satisfaction among PTC program leadership than among their 
college counterparts. But differences in question wording and willingness to be critical also may 
be at work. It would be useful to gather more direct measures of partnership functioning in the 
future. 


3.2 Recruitment and Admissions 


For each enrollment cohort, Year Up establishes recruitment targets it judges to be realistic for 
each PTC program while aiming for as much overall growth as is feasible. The study team was 
unable to locate an official statement of growth targets at the beginning of the four-year study 
period as a basis for assessing performance. In interviews, however, Year Up national staff and 
local PTC program leads told the team that meeting recruitment goals has been one of the 
program’s greatest challenges. 


PTC program enrollment increased fourfold between 2015 and 2018 (Exhibit 3-2, top panel). As 
discussed in Chapter 1, much of this growth arose from the addition of new programs rather 
than from increased enrollments within programs. By 2018, only four PTC programs were 
operating two or more learning communities.*' As planned, total enrollment across the eight 
core programs held steady at around 2,000. 


Although Year Up does not specify quotas for participants with particular characteristics, it 
aspires to serve a diverse population of young adults from low-income backgrounds who are 
motivated and in a position to benefit from the program experience. Year Up’s national leaders 


21 At about 40 participants per learning community and two recruitment cycles per year, two learning 
communities implies about 160 participants per year—the target for reaching break-even in the PTC 
financial model. 
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Exhibit 3.2: Fidelity of the PTC Programs on Recruitment and Admissions Dimensions of 
Implementation 


Performance Level 


High Fidelity Standard 
Enrollment Growth 
Steady growth in PTC 
enrollment 


Participant Profile 
Target young adults with 
manageable life 
challenges 


Reach underserved 
minorities 


Serve both genders 


Serve mixed age groups 


Mix of college experience 


Compatible goals for 
recruitment sources 


Measure 


Enrollment (calendar year) 


Number of risk factors (%) 
<2 risks 
2-4 risks 
5+ risks 


Race-ethnicity (%) 
African American 
Hispanic 
Asian/Pacific Island 
White 
Other/multiple race-ethnicities 


Gender = Female (%) 


Age below/above 20 (%) 
Age 20+ 


Prior college (%) 
Some prior college 


Targets for recruitment from college 


pipeline vs. community 
College partner's goals: 
College pipeline (%) 
General community (%) 
PTC leaders’ goals: 
College pipeline (%) 
General community (%) 


Actual share from general community 
(estimated by PTC Program Lead) (%) 


PTC Programs 


58.7 


51.4 


26.8 
73.2 


34.5 
65.5 


72.9 


Core Programs 


NA 


Sources: Demographic statistics are based on Year Up administrative data. Goals for recruitment by source 
represent averages across percentages cited as ideal by respondents to mid-2019 survey of PTC program leads and 


college liaisons. 


have emphasized that the PTC programs should recruit young adults whose characteristics are 
generally similar to those of their core program counterparts. 


PTC program admissions staff use a standardized rubric on a five-point scale to determine 
whether applicants have life challenges that are manageable with the type and level of supports 
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the PTC programs can provide. Year Up’s guidelines encourage programs to target applicants 
with a moderate level of risk—defined as having an average risk score of 2-4 points identified 
through an interview-based assessment that looks for 14 conditions that pose risks (each 
ranked 0-4 in seriousness). However, the guidelines do not stipulate specific performance 
targets. 


Slightly less than half (46 percent) of PTC program participants had moderate risks (two to four 
risk factors)—a level somewhat lower than the 56 percent of core program participants with two 
to four risk factors (Exhibit 3-2, panel bottom). Roughly similar fractions of the PTC and core 
participants had fewer than two and more than four risk factors (29 and 24 percent, 
respectively). 


Available demographic indicators show evidence of reasonably high levels of diversity among 
both PTC and core program participants. The majority of participants in both programs are from 
racial or ethnic minority groups. A higher proportion of participants in the PTC programs than in 
the core programs are African Americans (53 versus 43 percent). The PTC and core programs 
serve nearly identical mixes of men and women and of participants with and without previous 
college experience. The PTC programs have a markedly younger age profile than the core 
programs, reflecting that PTC programs recruit more heavily from high school pipelines than do 
the core programs. 


In struggling to meet challenging recruitment targets, it might seem natural for PTC program 
staff to work with their college partners to identify good candidates among college applicants. 
However, to date the college partnerships have been premised on the expectation that the PTC 
programs would be an important source of new recruits for their partner colleges, tapping young 
adults who otherwise would be unlikely to apply to college (see further discussion in Chapter 5). 


PTC program staff and partner college leads agreed that most recruits should come from the 
community (an average of 65 and 73 percent, respectively) rather than from college pipelines 
(Exhibit 3-2, bottom rows). The average PTC program lead estimated that 73 percent of actual 
recruits were from the general community. 


3.3. The Learning and Development (L&D) and Internship Phases 


Providing basic services during the L&D and internship phases of the program with high fidelity 
to the PTC design requires that program staff set performance expectations high and provide 
high levels of support to many participants. Year Up monitors L&D retention against its 83 
percent completion standard. It also expects local offices to consistently enforce Year Up 
behavior contracts. 


Slightly more than three-quarters (76 percent) of participants who enrolled in the PTC program 
during 2018 completed L&D—a retention rate somewhat below the 83 percent retention rate for 
core program participants (Exhibit 3-3). Nearly all participants in both the PTC and core 
programs were cited for at least one behavior infraction (97 and 99 percent, respectively). 
However, the PTC participants were less likely than their counterparts in core programs to 
receive 10 or more infractions (56 versus 72 percent). 
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Exhibit 3-3: Fidelity of the PTC Programs on Dimensions of the Learning and Development and 
Internship Services Provided 


Performance Level 


High Fidelity Standard 
High Expectations 
83% of Participants 
complete L&D 


Stringent contract 
enforcement 


90%+ of internship 
sponsors pay full rate 


90%+ of interns 
complete internships 


45%+ of internships 
convert to jobs 


75%+ of participants 
graduate 


High Support 
Strong exposure to Year 
Up learning community 


Supportive Year Up staff 


Supportive college staff 


Measure 


Participants completing L&D (%) 


Infractions recorded (%) 
Any infractions 
10+ infractions 


Employers pay full rate (%) 


Interns graduating (%) 


Internships converting to jobs (%) 


Graduation rate for all enrollees (%) 


Hours/week students spend with 
PTC program participants (avg.) 


PTC program leads rate staffing 
sufficient (%) 

For academic issues 

For non-academic issues 


College leads rate staffing sufficient 
(%) 

For academic issues 

For non-academic issues 


PTC program leads rate college 
staff sufficient (%) 


For academic issues 
For non-academic issues 


College leads rate college staff 
sufficient (%) 

For academic issues 

For non-academic issues 


PTC Programs 


76.1 


97.1 
56.3 


91.8 


85.7 


44.2 


66.3 


22.2 


46.7 
73.3 


78.5 
92.9 


36.7 
20.0 


64.3 
57.1 


Core Programs 


83.0 


98.7 
72.2 


NA 


90.2 


47.8 


15.5 


NA 


NA 
NA 


Sources: Administrative statistics provided by Year Up, mid-2019 survey of PTC program leads and college liaisons. 
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One possible reason for fewer infractions among PTC program participants is that PTC staff 
spend less time with participants in a typical week than do their counterparts in the core 
programs. Local PTC program leads estimated that students spend 22 hours a week with other 
participants and staff as a learning community. In contrast, the core program involves full-time 
attendance at stand-alone programs. As a result, core participants spend close to 35 hours a 
week with program staff, and staff thus have more opportunities to observe participants’ 
behavior. 


On the “high support” front, PTC program leads were more likely to assess Year Up supports as 
sufficient for resolving non-academic challenges than for addressing academic issues (73 
versus 47 percent).2* Fewer PTC than college leads rated college supports as sufficient for 
either issue type.”° College liaisons were generally more likely to rate supports as sufficient. 
They tended to rate PTC program supports more highly than college supports—even when it 
came to addressing academic issues. 


Year Up aims to collect its full payment rate for internships for 90 percent of internships in both 
its PTC and core programs. The PTC programs have consistently met that standard—the 
average collection rate in 2018 was 92 percent (Exhibit 3-3, top panel). High expectations for 
the internship phase also include achieving a 90 percent completion rate and, for those 
completing internships, having at least 45 percent of internships convert to jobs with the 
employer that sponsored the internship. The PTC programs came close to meeting these 
targets in 2018, with an 86 percent L&D completion rate and a 44 percent conversion rate. The 
core program scored only slightly higher on these metrics (90 and 46 percent, respectively). 


Year Up maintains a 75 percent standard for overall PTC program graduation—the same as for 
the core program. Overall, PTC programs have fallen about 10 percentage points below this 
standard. Graduation rates for PTC and the core program were 65 and 76 percent, respectively, 
in 2018 (Exhibit 3-3, top panel, bottom row).74 


22 Based on the study team’s summer 2019 survey, these statistics summarize responses to items asking 
local PTC program leads and their counterparts at the partner colleges to gauge the adequacy of 
supports provided by the PTC programs and by the college over the course of the program. Year Up 
also monitors satisfaction through regular surveys of participants and, during the internship phase, of 
employers. Although shifts in customer satisfaction can be helpful in identifying problems requiring 
attention, the survey did not probe such issues; the study team judged that the survey responses 
would likely not be sufficiently reliable and valid to support analyses of implementation fidelity. 

As noted earlier, insufficient college support services prompted Year Up to rethink and increase levels 
of PTC social services staffing. 


Completion rates for both programs changed little during the four-year study period (not shown in 
Exhibit 3-4). For the PTC programs, rates were 62, 67, 65, and 65 percent for the years 2015-2018, 
respectively. The corresponding figures for the core programs were 77, 76, 76, and 75 percent. 
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3.4 Post-Program Outcomes 


Year Up standards include a series of targets for desired outcomes for program graduates 
measured at four and 12 months after program graduation (Exhibit 3-4). Standards for most 
outcomes were the same for the PTC and core programs: (1) an average wage among 
employed graduates of at least $16; (2) at least 85 percent of graduates employed or in school; 
and (3) at least 70 percent of graduates employed in a full-time, Year Up—related job. 


Exhibit 3-4: Outcomes for 2018 Graduates of the Year Up’s PTC and Core Programs 
Performance Level 


PTC 

High Fidelity Standard Measure Programs Core Programs 
Outcomes at 4 months post- 
program 
$16+ average starting wage Starting hourly wage ($/hr) 18.70 20.90 
85%+ employed/in school Graduates employed/in school (%) 86.1 87.8 
70%+ employed in full-time, YU- | Graduates employed in full-time, YU- 62.4 76.3 
related jobs related jobs (%) 
75% in school (PTC programs Graduates enrolled in school (%) 34.7 10.9 
only) 

Total graduates (N) 973 1,543 
Outcomes at 12 months post- 
program 
70%+ contacted at the 12- month Graduates contacted (%) 70.6 71.2 
follow-up 
$16+ average starting wage Average starting wage ($/hr) 19.10 21.90 
85%+ either employed/in school Graduates employed/in school (%) 90.0 85.6 
70%+ employed in full-time, YU- Graduates employed in full-time, YU- 66.5 711 
related jobs related jobs (%) 
PTC only: 75% in school or Graduates enrolled in school (%) 479 24.9 
received credential 

Graduates contacted (N) 687 1,099 


Sources: Administrative statistics provided by Year Up. 


To reinforce the importance of extended follow-up, standards also included achieving at least a 
70 percent contact rate with program completers 12 months after graduation. 


Reflecting the PTC program’s focus on college, beginning with the cohort scheduled to graduate 
from the program in July 2016, Year Up applied additional standards to PTC programs. These 
standards are that (1) four months after program graduation, at least 75 percent of PTC 
graduates should be enrolled in school; and (2) 12 months after graduation, 75 percent should 
be enrolled in college or have earned a credential. 


The study team measured performance on these metrics for 2018 graduates of Year Up’s PTC 
and core programs (Exhibit 3-4). Average starting wages post-program for both groups of 
graduates were well above the $16 standard at both the four-month follow-up ($18.70 and 
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$20.90 for the PTC and core program graduates, respectively) and 12-month follow-up ($19.10 
and $21.90 for the PTC and core program graduates, respectively).° 


PTC program graduates were markedly less likely than core program counterparts to be 
working in full-time, Year Up—related jobs four months after graduating from the program (62 
and 76 percent, respectively). However, by the 12-month follow-up, the gap between graduates 
of the two programs had narrowed considerably (67 and 71 percent, respectively). 


At both time points, PTC program graduates were much more likely than core program 
graduates to be enrolled in school (35 versus 11 percent at the four-month follow-up, and 48 
versus 25 percent at the 12-month follow-up). Neither program came close to meeting Year 
Up’s standard for PTC that 75 percent of graduates should be enrolled in school (Exhibit 3-4). 


As discussed in Chapter 2, effective July 2019, Year Up dropped the 75 percent school 
enrollment target for PTC graduates. In its place, it introduced a two new standards. First, at 
graduation, 100 percent of PTC and core program graduates should have prepared an 
actionable career plan specifying near- and longer-term education goals and planned actions to 
achieve them. Second, at the 12-month post-graduation follow-up, a majority of graduates 
should be “on track” with their plans. As of mid-2019, Year Up had not decided how it will 
measure performance relative to the latter standard. 


3.5 Summary 


Evidence in this chapter shows that Year Up has made substantial progress in implementing the 
PTC program but also that performance gaps remain. The program has achieved high fidelity on 
standards for key design elements such as lowering per-participant costs relative to those for 
the core programs, operating with a leaner staff, recruiting youth fitting the target profile, 
collecting full revenue for internships, and having high proportions of graduates in either well- 
paying jobs or school. In most instances, where the PTC program has fallen short of 
performance targets, the gaps seem surmountable. For example, Mini-Study #1 showed that a 
10 percentage point increase in retention to the internship phase should be feasible at current 
staffing levels if local PTC programs adopt strategies similar to the ones tested in this project. 


It is important to recognize that Year Up has set the bar unusually high, in holding PTC 
programs to the outcome standards of its core program while expecting them to operate at 
substantially lower costs. Compared to other workforce programs, a year of full-time training, 
support services, and internships is very intensive. Most workforce and college programs would 
be happy to achieve a 65 percent completion rate and cover more than three-fifths of their costs 
through revenue provided by employers. 


25 Year Up National staff noted that differences in local wage rates in core and PTC markets are a likely 
contributor to these differences in average wages. 
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4. Early Impacts on Education and Employment 


This chapter presents initial results from a small randomized controlled trial (RCT) designed to 
measure the overall impacts of several Professional Training Corps (PTC) programs. Over a 
two-year period beginning in July 2017, staff at three PTC programs enrolled four cohorts of 
program applicants (554 young adults) in the study. Sample members randomly assigned to the 
treatment group were encouraged to enroll in PTC, whereas those assigned to the control group 
were not allowed to enroll but could receive training and services otherwise available in the 
community. 


This chapter presents initial evidence for program impacts on college enrollment, employment, 
and earnings. Analyses draw on administrative data on college and employment outcomes and 
cover the program and immediate post-program periods (four calendar quarters in the program, 
one following it). Subsequent reports will analyze impacts over a longer follow-up period.”° 


The study’s early findings mirror expectations based on the Year Up program logic model and 
the pattern of results from a large-scale RCT of Year Up’s core program (Fein & Hamadyk, 
2018). The current study finds that PTC treatment group members had significantly higher rates 
of college enrollment during the program year. The gains were especially large during the initial 
six-month Learning & Development (L&D) phase of the program, when all participants are 
required to enroll full-time in courses at partner colleges. 


PTC programs require full-time participation during both the L&D and internship phases. 
Consistent with this requirement, the treatment group had significantly lower employment and 
earnings than the control group during the first four quarters following enrollment. Similar to the 
findings for the study of Year Up’s core programs, earnings reductions were larger during the 
L&D phase (Q1-Q2) than during the internship phase (Q3-Q4). By the start of the post-program 
period (Q5) they had disappeared. 


The chapter begins with a description of the research questions guiding the study (Section 4.1). 
It then discusses site selection, sample design and enrollment procedures, characteristics of the 
sample, and analyses (Section 4.2). Section 4.3 presents the findings, and Section 4.4 provides 
a summary discussion. 


4.1. Hypothesized Early Impacts 


This study’s hypotheses flow from three key steps of the PTC program model: (1) During L&D, 
the model seeks to boost English skills, technical skills in an occupation, and professional skills 
needed in the workforce. (2) The program provides workplace-based learning opportunities. 

Participants hone technical skills and build career awareness and professional networks during 
full-time, six-month internships (often while taking additional college courses). (3) The program 
provides job search and related career supports to help graduates make successful transitions 
to work and continuing education. As discussed in Chapter 2, the local PTC programs place 


26 Future planned reports are being funded under a separate grant from Arnold Ventures. 
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greater emphasis on full-time employment immediately after the program than on continuing 
education. 


RCT findings for the core program are helpful in anticipating the expected short-term impacts of 
the PTC programs. Because the core and PTC programs follow the same basic logic (see 
Chapter 1), a finding of similar early impacts may signal that the PTC programs are on track for 
longer-term success. The core program substantially increased college enrollment rates during 
the program—that is, the L&D and internship phases—largely through co-enrollment at local 
partner colleges, and it reduced the rates in the year following the program. The core program 
had the opposite effect on employment and earnings, substantially reducing them during the 
program and increasing them starting in the first post-program quarter (Fein & Hamadyk, 2018, 
Exhibits 6-2 and 6-6). 


The impacts of the PTC programs should follow similar patterns during the first (program) year. 
Accordingly, this chapter tests the hypotheses that during the first year following enrollment the 
PTC program will (1) increase the probability of college enrollment and cumulative months of 

full-time college enrollment and (2) reduce the likelihood of employment and average earnings. 
The PTC program model clearly aims to increase participants’ earnings following the end of the 
program, but post-program expectations for college enrollment are somewhat more ambiguous. 


To test these hypotheses, the early analyses reported here use outcome data for the first five 
calendar quarters of follow-up after random assignment. To put follow-up time in context of PTC 
program time, Quarter 1 is the first quarter in which young adults assigned to the treatment 
group are in L&D. Quarter 2 is a period during which they finish L&D and transition to 
internships. Quarter 3 is a period when they are in internships. Quarter 4 is the period when 
treatment group members complete their internships and leave the program. Finally, Quarter 5 
is the only exclusively post-program quarter with data available at the time of these analyses. 


4.2 Evaluation Design and Analysis Plan 


The study used an experimental design that involved randomly assigning conditionally eligible 
PTC program applicants to either a treatment or control group. The former had the opportunity 
to enroll in the PTC program, whereas the latter were not allowed to enroll but could pursue 
similar education and training and receive similar services from other providers in the 
community (including the partner college). Randomization helps to ensure that estimated 
impacts (differences in outcomes between the groups) reflect true causal effects of the program 
rather than differences in the characteristics of treatment versus control group members. 


4.2.1 Site Selection, Sample Recruitment, and Random Assignment 


The study team worked closely with Year Up national staff to identify three PTC programs to 
participate in the study and to develop procedures for sample recruitment and randomization. 


Site selection. In selecting sites, we sought programs that had the capacity to over-recruit by at 
least 50 percent in order to generate a control group, were operationally mature, and had good 
performance histories (see Appendix Exhibit B-1). Characteristics of young adults in the three 
selected sites were very similar to those of all participants in PTC programs (Appendix Exhibit 
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B-2). Slightly more than half of all PTC participants are male and aged 20 or older. Most 
identified as black/African American or Hispanic, and roughly half had at least some prior 
college experience. 


The three programs’ statistics for key program performance outcomes are also very similar to 
PTC programs overall. For example, 79 percent of PTC program participants in the study sites 
and in the non-study sites completed L&D, and 67 percent graduated from the program. 


Within the study sample, there were some differences across the three participating programs.”’ 
For example, the percentage of participants in the study sites who identified as black or African 
American ranged from 27 to 85 percent; the percentage who were female ranged from 31 to 56 
percent; and the percentage with zero or one risk factor ranged from 27 to 69 percent (Appendix 
Exhibit B-2). L&D completion rates in the study sites ranged from 77 to 89 percent, and overall 
graduation rates ranged from 57 to 85 percent. 


Recruitment. Local PTC offices recruit on an ongoing basis throughout the year, but they enroll 
new cohorts only twice a year. Recruitment and enrollment in the study extended from mid-2017 
to mid-2019. Over this period, the study enrolled four cohorts of conditionally eligible program 
participants.”° 


As part of the application process, PTC program staff provided information about the study to 
applicants who were identified as conditionally eligible for the program.” This information 
explained that admission to the PTC program would be through a lottery of those eligible 
applicants who gave informed consent to participate in the study.*° 


Random assignment. The study randomized conditionally eligible applicants within specified 
groups, or “blocks.” Blocking helped to ensure that each program office received enough 
treatment group members to fill slots allocated to particular training tracks. Local program staff 
used a web-based lottery tool to assign eligible applicants to the treatment or control group. This 
allowed the study team to flexibly adjust probabilities of assignment to the treatment group as 
circumstances warranted. Over this enrollment period, the resulting probability of being 
assigned to the treatment group varied from 50 to 85 percent.*' 


In total, 552 conditionally eligible youth enrolled in the study sample—3839 in the treatment 
group and 163 in the control group (Exhibit 4-1). Seventy-four (74) percent of those assigned to 


27 As in other chapters, the analysis identifies offices by number to protect anonymity. 

28 Site 1 enrolled sample in the first and second enrollment cohorts, Site 2 enrolled sample in the second 
through fourth cohorts, and Site 3 enrolled sample in the third and fourth cohorts. 

29 Conditional eligibility was based on information available at the time of application. Programs 
sometimes uncovered information during the PTC enrollment and college registration process that 
showed the applicants to be ineligible. 

30 This information included an advisory that applicants assigned to the control group by lottery would be 
embargoed to enroll in the PTC program for three years. 

31 See Exhibit B-3 for details on the assignment blocks and assignment probabilities by site and 
Exhibit B-4 for further details on the recruitment and randomization process. 
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the treatment group enrolled in the PTC program, 21 percent opted (dropped) out prior to 
enrollment, and 6 percent were found to be ineligible after randomization. 


Exhibit 4-1: Random Assignment and Enrollment in the Study Sample 


Applicants Conditionally Accepted? 


N =552 


Control Group Random 
N = 163 | Assignment 
I 


Treatment Group” 
N = 389 


Participated 
N = 286 


= {ab} 
6 2 
se =) 
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Source: Authors’ tabulations of Year Up administrative data. 


4 Conditional acceptances were withdrawn most often for academic issues, such as determining applicants did not 
have basic skills required by the college, failure of applicants to take a required academic placement test, or being 
placed on academic probation (when the applicant was already enrolled in the college prior to applying to the PTC 
program). It was not possible to defer randomization until applicants finalized their college registration, as that would 
have disrupted intake procedures of both the PTC programs and the colleges. 


> Seven (7) treatment group members retained in the sample initially opted out of the program but subsequently 
enrolled in a later cohort. 


4.2.2. Data Sources and Alignment with Program Time 


The impact analysis draws on data from three sources. Year Up administrative data provide 
measures for pre-random assignment (“baseline”) characteristics that were used as covariates 
in regression models to estimate impacts (see Section 4.2.4 below). The National Student 
Clearinghouse (NSC) provides data on college enrollment outcomes. The National Directory of 
New Hires (NDNH) provides data on employment and earning outcomes. 


Each of the three data sources provided high coverage of its focal outcomes. Year Up 
administrative data are primarily gathered through program applications. Thus, they are 
available for the full sample and have very low rates of item non-response. NSC and NDNH 
data are available for 94 to 96 percent of the sample, respectively—with small missing fractions 
resulting from errors in the identifiers used in matching (generally Social Security number). 


The NSC records reflect spells of college enrollment based on information participating colleges 
provide to the NSC on an ongoing basis. NSC coverage is extremely high for public two- and 
four-year colleges and for private non-profit four-year colleges. It is somewhat lower for non- 
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profit two-year colleges and substantially lower for for-profit colleges (National Student 
Clearinghouse, 2018). NDNH records capture the total earnings in every calendar quarter 
received by workers at reporting employers. Certain types of employment are not covered. 


Analyses are based on July 2019 and February 2020 data extracts from NSC and NDNH, 
respectively. The difference in timing helps to equalize the duration of available follow-up, given 
the typical two-quarter lag in employer-provided wage records reaching NDNH. Analyses cover 
five calendar quarters following enrollment in the study, which is the start of L&D for members of 
the treatment group. Quarter 5 is the first full post-program calendar quarter.*° 


4.2.3. The Study Sample 


Characteristics of RCT sample members closely resembled those of the overall PTC participant 
populations in the three study offices, as described in Section 4.2.1. Slightly more than half of 
the sample was male, and nearly one-third identified as black/African American or Hispanic 
(Exhibit 4-2). Fewer than 20 percent had zero or one assessed risk at intake; 57 percent had 
four or more risks. Nearly 70 percent had attended college full-time at some point, but fewer 
than 3 percent had received a college credential. Sixty-three (63) percent were employed three 
calendar quarters before enrollment, and they earned an average of $2,529 in the calendar 
quarter prior to random assignment. 


Randomization produces treatment and control groups whose baseline characteristics differ 
only by chance. The two groups in this study look reasonably similar and do not differ 
statistically for most of the background characteristics examined (Exhibit 4-2). One exception is 
that treatment group members were somewhat more likely to be African American and 
somewhat less likely to be white (67 and 14 percent, respectively) than control group members 
(62 and 22 percent, respectively). Although employment and earnings seem somewhat higher in 
the treatment than in the control group, the difference was not statistically significant. 


Further investigation revealed no defects in the random assignment procedures, suggesting that 
these differences are likely due to chance. Including race-ethnicity and baseline earnings as 
covariates in the regression models used to estimate program impacts minimizes any potential 
bias in the findings due to these chance differences (see Section 4.2.4). 


82 Under the federal Unemployment Insurance Compensation laws governing wage reporting, for 
example, employment in the informal economy is mostly not covered, and earnings from self- 
employment and work as an independent contractor or at a gig job is subject to varying requirements in 
different states. 

33 See more details on the alignment of calendar and program time for the individual study sites and 
enrollment cohorts in Appendix Exhibits B-6 and B-7. 
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Exhibit 4-2: Characteristics of the Study Sample Members at Baseline 
Random Assignment 


SEMIS 
Treatment Control Group 

Characteristic eT oli) Group Difference p-Value 
Gender Male (%) 53.5 50.5 56.6 -6.1 2021 
Race-ethnicity (%) .0784* 

Black or African American 64.3 61.6 67.1 -5.5 

Hispanic or Latino 17.5 16.0 19.1 -3.1 

White or Another Race 18.1 22.4 13.8 8.6 
Risk Factors (%)@ 3482 

0-1 17.6 15.5 19.9 -4.4 

2-3 25.1 27.7 22.4 5.3 

4-5 23.7 21.8 25.7 -3.9 

6+ 33.6 35.1 32.0 3.0 
Factors of Success (4-item average® 0.98 0.98 0.99 0.01 8933 
IT or Software Develop. Interest (%) 51.3 51.7 50.9 0.9 8582 
Prior Full-time College (%) 1330 

None 28.7 31.2 31.3 -0.1 

<1 year 44.9 43.4 40.3 3.1 

1+ years 26.3 25.4 28.4 -3.0 
Some College Credential (%) 2.3 3.2 14 18 1724 
Employment in the 3° quarter prior to 62.6 64.9 57.3 76 1189 
enrollment (%)° 
Earnings in the 3"¢ quarter prior to 2,635 2,636 2,214 361 2191 
enrollment ($)¢ 
Sample Size 552 389 163 552 


Source: Authors’ tabulations of administrative data from Year Up, the National Student Clearinghouse, and National 
Directory of New Hires. 


Notes: Data are weighted to adjust for different probabilities of assignment to treatment and control groups across 
cohort and office. The p-values give the probability of observed associations between the characteristic and treatment 
group status based on chi-squared (for categorical characteristics) and t (for metric characteristics) statistics. 


Statistical significance: * = .10 level, ** =.05 level, and *** =.001 level on two-tailed tests. 


4 The risk factors are measured using a risk inventory completed at application. A higher risk score indicates more 
challenges that may impede success in the program, such as financial issues, housing issues, or a history of 
substance abuse. 


>The Factors of Success score summarizes staff ratings of each applicant across four items (applicant’s critical 
thinking, persistence, interpersonal awareness, and striving to learn). Its values range from 0 to 1, where 0 represents 
the most favorable score. 


© NSC and NDNH only: Quarter 3 prior to enrollment was used as the baseline control measure because it is the 
latest quarter of pre-enrollment data that would be unaffected by contact with Year Up, This is because Year Up 
enrolls new cohorts every six months, but it accepts applications continuously. 


4.2.4 Analytic Approach 


The study team estimated impacts by pooling results across sites and then comparing means 
for treatment and control group members. Analyses use regression models to control for the 
baseline characteristics in Exhibit 4-2 and a series of binary indicators for site, cohort, and 
treatment status.*4 In addition to attenuating any effects of chance differences arising during 
random assignment, regression adjustment can increase the statistical power of the analysis. 


34 See Appendix Exhibit B-5 for additional details. 
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The tables in this chapter reporting impact findings (Exhibits 4-3 and 4-4) include several key 
statistics for each outcome measure and time period: regression-adjusted means for the 
treatment group, unadjusted means for the control group, the estimated impact (difference in 
treatment and control group means), and the standard error and p-value for the impact 
estimate.°° 


As noted earlier, only 74 percent of treatment group members participated in the PTC program. 
The main results in this chapter apply to the full treatment and control groups—i.e., including 
treatment group members who staff initially identified as conditionally eligible but who later did 
not enroll. These estimates—referred to in the literature as “Intent to Treat” impacts—will tend to 
understate impacts sample members who actually did participate in the program. A simple 
adjustment yields estimated impacts for those young adults who actually participated.*° 
Appendix Exhibits B-9 and B-10 report these estimates. 


4.3. Findings 


The early impact findings presented in this section are broadly consistent with the PTC program 
logic model and with the findings for Year Up’s core program.*’ The following sections describe 
findings for the program period and the immediate post-program periods, in turn. 


4.3.1. Impacts during the Program Period 


Exhibit 4-3 shows impact estimates for college enrollment and employment and earnings during 
the first four quarters following program enrollment. These quarters roughly cover the L&D 
phase (Q1-Q2) and internship phase (Q3-Q4) of the PTC program. 


College enrollment. As expected, the treatment group had significantly higher rates of college 
enrollment and more months of full-time enrollment during the program period than did the 
control group (Exhibit 4-3, top panel). The gains were especially large during the L&D phase. 
For example, in the first quarter following enrollment (Q1), 79 percent of the treatment group 
was enrolled in college, compared with only 37 percent of the control group—a statistically 
significant 42 percentage point difference.*® 


College enrollment among the treatment group declined substantially in successive quarters as 
program participants began transitioning into full-time internships. Only 49 percent of the 


35 The standard error measures the sampling error in the estimate. The p-value is the probability that the 
estimated impact would have been observed by chance even if the program had no actual impact on 
the outcome. 


The so called “no-show adjustment” divides the impact estimated for the full study sample by the 
participation rate (Bloom, 1984). The resulting estimates are referred to as “treatment-on-the-treated” 
(TOT) impacts. Given a 74 percent participation rate, TOT impacts will be 30 to 40 percent larger than 
impacts measured for the full sample. (Impacts for the full sample are called “intent-to-treat, or ITT, 
impacts.) 

See Fein and Hamadyk (2018). 


One of the three PTC programs participating in the study recruited applicants largely from its local 
college partner. As a result, 85 percent of its control group was enrolled at the college during Q1. In 
comparison, less than a third of control group members in the other two sites were enrolled in college 
during Q1. With only three sites in the study, the impact estimates are sensitive to this variation. 
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treatment group was enrolled in college anytime during Q3, compared with about 35 percent of 
the control group. By Q4, the two groups’ enrollment rates were nearly identical (22 versus 23 
percent). 


Exhibit 4-3: Estimated Impacts During the Program Period (Quarters 1-4) 


(1) (4) Estimated Mean Program Impact 
Treatment Control 


Group Group Standard 
Outcome Mean? Mean? (Col. 1 - Col. 2) Error eAVELU 
College Enrollment (Cohorts 1-3 for Q1-3; Cohorts 1-2 for Q4) 
Any College Enrollment (%) 


Quarter 1 79.2 37.0 42.2 ** 48 <.0001 
Quarter 2 61.0 33.9 Zt, 49 <.0001 
Quarter 3 49.0 34.8 14.1 ** 48 0.0032 
Quarter 4 21.7 23.0 -1.3 49 0.7878 
Cumulative Months of Full-Time Enrollment 
Quarter 1 3.4 1.3 oA oF 0.2 <,0001 
Quarter 2 4.5 2.1 24 ** 0.3 <.0001 
Quarter 3 5.0 2.7 2a 3 0.3 <.0001 
Quarter 4 5.0 2.2 2.8: ** 0.4 <.0001 
Employment & Earnings (Cohorts 1-3 for Q-1-3; Cohorts 1-2 for Q4) 
Any Employment (%) 
Quarter 1 56.8 67.4 10.6 * 4.9 0.0312 
Quarter 2 52.6 77.2 -246 *** 45 <.0001 
Quarter 3 54.2 74.2 -20.0 *** 49 <.0001 
Quarter 4 66.1 73.2 -7.2 5.3 0.1756 
Total Earnings ($) 
Quarter 1 1,688 3,120 1432 ** 334 <.0001 
Quarter 2 1,515 3,135 -1620 ** 342 <.0001 
Quarter 3 1,766 3,571 1805 ** 359 <.0001 
Quarter 4 2,387 3,158 7m * 436 0.0112 
Sample Size (Cohorts 1-3 / Cohorts 1-2) 
College Outcomes 302/229 123/101 425/330 
Employment Outcomes 292/221 119/97 411/318 


Source: College enrollment data from the National Student Clearinghouse. Employment and earnings data from the 
National Directory of New Hires database. 


Notes: Estimation methods are described in Appendix Exhibit B 4-8. Data have been weighted to account for 
differential probabilities of assignment to the treatment or control group. 


Statistical significance: * = .10 level, ** = .05 level, and *** = .001 level on two-tailed tests. 
4 Treatment group means and the estimated impacts of the program are regression adjusted. 
> Control group means are not regression adjusted. 
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Much of the decline in college enrollment among PTC participants between Q1 and Q2 was 
associated with program attrition, whereas the decline between Q2 and Q3 reflects the fact that 
not all PTC programs required college enrollment during the internship phase.*? 


Over the program period, the treatment group spent more than twice as many months enrolled 
full-time in college than did the control group (5.0 versus 2.2 months)—a difference that is 
statistically significant. Two-thirds of this gain accrued by the end of Q1, when participants were 
enrolled in school full-time as part of their L&D experience. 


Employment and earnings. The treatment group had significantly lower employment rates and 
earnings throughout the first four quarters following program enrollment (Exhibit 4-3, bottom 
panel). During Q1, when participants were enrolled in college full-time, the employment rate for 
the treatment group was 11 percentage points lower (statistically significant) than that for the 
control group. By Q2, the control group had increased its employment rate by 10 percentage 
point, while the treatment group had decreased its rate by 4 percentage points, creating a 25 
percentage point difference (53 versus 77 percent). 


This treatment-control employment gap declined slightly in Q3 and considerably more in Q4 as 
treatment group members began completing the program. By Q4, the impact was not 
statistically significant. The fraction of treatment group members employed increased by 

12 percentage points between Q3 and Q4. 


Over the first three quarters following program enrollment, the treatment group’s average 
quarterly earnings were less than half those of the control group. For example, in Q3, when 
many treatment group members were in full-time internships, treatment group earnings 
averaged $1,766, compared to $3,571 for the control group. These negative impacts over the 
first three quarters are somewhat larger than those observed in the evaluation of Year Up’s core 
program (Fein & Hamadyk, 2018, p. 72), an especially notable result given that far fewer PTC 
than core treatment group members actually participated in the program.*° 


PTC program stipends provided only modest offsets to the treatment group’s forgone earnings. 
For example, in Q1 (the only quarter that overlaps completely with L&D), stipends averaged 
$345 (compared to a $1,432 earnings drop). In Q3 (the only quarter that overlaps completely 
with the internship phase), stipends averaged $1,129 (compared to a $1,805 drop in 
earnings).*" 


In Q4, when participants were transitioning to post-program jobs, the treatment-control earnings 
gap decreased to -$771, a figure that once adjusted for the fact that 26 percent of the treatment 
group did not participate in the program, is also somewhat lower than the earnings gap for 


39 Program attrition during L&D was frequently associated with poor performance in college courses. See 
Chapter 2 for a discussion of Mini-Study, which tested responses to this issue. 

40 Whereas 74 percent of PTC treatment group members participated in the program, fully 96 percent of 
core treatment group members did so. 

41 These average stipend values treat the stipend as $0 for the approximately 26 percent of the treatment 
group that did not enroll in the PTC program. Excluding those sample members, the average stipend 
values in Q1 and Q3, respectively, were approximately $470 and $1,529. 
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participants in the core program during the fourth quarter following enrollment (-$402, Fein & 
Hamadyk, 2018). 


4.3.2 Impacts Just After the Program Period 


Data available in time for this report allow analysis only for outcomes into the first post-program 
quarter (follow-up Q5) for the first two cohorts enrolled in the study sample. Exhibit 4-4 
summarizes the results. 


Exhibit 4-4: Estimated Impacts on College Enrollment, Employment, and Earnings Post-Program 
(Quarter 5) 


tc) (4) Estimated Mean Program Impact 
Treatment Control (6) 


OTN Cero T=) Group Group Impact Estimate Standard 
(Program Period) Mean Mean? (0X0) His Ia Oe) 4) Error 
College Enrollment Outcomes (Cohorts 1-2) 
Any College Enrollment (%) 


Quarter 5 19.2 23.0 -3.8 5.2 0.4631 
Cumulative Full-time Months Enrolled in College 

Quarter 5 5.3 2.6 28 ** 0.4 <.0001 
Employment & Earnings Outcomes (Cohorts 1-2) 
Any Employment (%) 

Quarter 5 77.8 78.2 -0.5 49 0.9252 
Total Earnings ($) 

Quarter 5 4,677 4,292 385 470 0.4138 
Sample Size (Cohorts 1 & 2) 

College Outcomes 229 101 330 

Employment Outcomes 221 97 318 


Source: College enrollment data from the National Student Clearinghouse. Employment and earnings data from the 
National Directory of New Hires database. 


Notes: Estimation methods are described in Appendix Exhibit B 4-8. Data have been weighted to account for 
differential probabilities of assignment to the treatment or control group. 


Statistically significance: * = .10 level, ** = .05 level, and *** = .001 level on two-tailed tests. 
4 Treatment group means and the estimated impacts of the program are regression adjusted. 
> Control group means not regression adjusted. 


College enrollment. By the fifth quarter following enrollment, when treatment group members 
all typically have completed PTC, college enrollment rates for the treatment and control groups 
converge.*? Nineteen (19) percent of the treatment group and 23 percent of the control group 
were enrolled in college in Q5—a difference that is not statistically significant (Exhibit 4-5, top 
panel). 


Reflecting the build-up of earlier positive impacts, the average cumulative number of full-time- 
equivalent college enrollment months remained 2.8 months higher for treatment than for control 


42 Only the first and second enrollment cohorts have data for Q5; they include only sites 1 and 2. 
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group members. By the end of Q5, treatment group members averaged 5.5 months of full-time 
enrollment, compared to 2.6 months for the control group. 


Employment and earnings. A nearly identical fraction of treatment and control group members 
(78 percent) had some earnings (i.e., were employed) in Q5 (Exhibit 4-4, bottom panel). The 
impact on average earnings, which had been negative in the prior quarter (-$771), turned 
positive. Although the estimate (+$385) was heading in the desired direction, it was not 
statistically significant. 


The turnaround occurred because average earnings increased more for treatment than for 
control group members between Q4 and Q5. Earnings increased 70 percent for the former but 
only 20 percent for the latter. Time will tell whether earnings differences grow large enough to 
reach statistical significance over a longer follow-up period. 


4.4 Discussion 


Year Up’s local PTC programs produced moderately large increases in college enrollment 
during the program’s L&D phase and smaller increases during its internship phase. 
Notwithstanding PTC’s more active college partnerships, its enrollment impacts were about the 
same as those of Year Up’s core program.*° 


One reason PTC college enrollment rates were not higher is that control group members 
enrolled in college at rates higher than they did in the core program study. The difference may 
reflect PTC programs’ location on college campuses where they draw more heavily from young 
adults interested in attending local colleges. Furthermore, one study site recruited heavily from 
students already enrolled in the partner college, as noted earlier. 


Another possible reason impacts on college enrollment were not larger is that more young 
adults assigned to the PTC treatment group withdrew prior to the start of classes than withdrew 
from the core program. Also, Year Up generally did not require PTC participants to maintain 
college enrollment during the internship phase (though some college partners counted the 
internship and associated weekly seminar as a credit-bearing college course). Some programs 
also encouraged interns to enroll in additional online and night classes during internships. 


This early analysis of employment and earnings impacts of PTC mirrors the pattern of findings 
from the evaluation of Year Up’s core program. Like the latter, the PTC treatment group had a 
substantially lower employment rate than the control group during the program (Q1 to Q3). 
However, by Q4 the negative impact had shrunk considerably, and by Q5 (when none of the 
treatment group was still in the program) it had disappeared. 


For PTC, during the year following program enrollment (Q1-Q4), the treatment group’s average 
earnings was less than half the average earnings of the control group ($5,367 versus $12,984). 


43 See Fein and Hamadyk (2018). 
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The treatment-control gap of $7,617 is about 40 percent larger than was seen for the core 
program, after adjusting for different rates of program participation.** 


As in the core program study, PTC treatment group members’ employment and earnings 
increased substantially as they completed the program. By Q5—when all treatment group 
participants typically have completed the program—impacts on average earnings turned 
positive, though not statistically significant. Longer-term follow-up is needed to see whether and 
by how much impacts grow. Planned longer-term analyses will estimate impacts on college and 
employment outcomes for up to three years following program enrollment for the full sample. 


44 The estimated average earnings impact for PTC program participants over Q1-Q4 is -$7,617 
(summing the quarterly averages; see Appendix Exhibit B-9). The comparable figure for the core 
program is -$5,440 (Fein & Hamadyk, 2018, p. 72). 
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5. Perspectives on Implementation Going Forward 


This chapter assesses perspectives on implementation of the Professional Training Corps 
(PTC) program at the end of the study period. Analyses are based on interviews and an online 
survey with key program stakeholders conducted between April and July of 2019. 


The main data source is a set of interviews with Year Up national leaders (nine respondents), 
local PTC program leads (four respondents), and college and employer partners (two and three 
respondents, respectively). The interviews covered major goals and experience with 
implementation to date. The discussion also draws on interviews with an additional nine national 
and local Year Up staff that focused primarily on issues addressed in the mini-studies (Chapter 
2) but that also touched on wider issues, and on results from an online survey described in 
Chapter 3.*° 


Each of the chapter's eight sections summarizes perspectives on an important implementation 
challenge. Topics include alignment on PTC’s goals (Section 5.1), challenges performing well at 
a lower cost (Section 5.2), meshing with college missions and operations (Section 5.3), 
addressing recruitment difficulties (Section 5.4), priorities for improvement (Section 5.5), 
program components most critical to success (Section 5.6), requirements for scaling up (Section 
5.7), and gathering evidence “to improve and to prove” PTC programs’ effectiveness (Section 
5.8). 


5.1. Alignment on PTC’s Goals 


In any large, multi-site initiative involving diverse stakeholders, goals and performance 
expectations can vary across sites and categories of stakeholders. Such variation is normal, 
given differing local conditions and institutional imperatives, and it can be healthy when 
programs are able to accommodate the multiple goals. This section describes the goals of Year 
Up national and local leaders for the PTC programs and compares them with the goals of PTC 
college partners. 


When asked about their goals for PTC program operations, Year Up and program leads cited 
smoothly functioning program activities, close coordination with college operations, and meeting 
performance targets. One local PTC lead explained that visitors to well-functioning program 
offices would see participants and staff in place, engaged, passionate about their work, and 
actively involved in other campus activities. They might see college personnel on site 
participating in learning community activities. They also might see or hear that PTC staff were 
working closely with college administrators and instructors on general planning, recruitment and 
admissions, curriculum and instruction, internships, and career planning. 


Year Up had specific numeric goals for PTC program performance. As discussed in Chapter 3, 
these goals include consistently hitting targets for retention, reducing costs to reach break-even 


45 As described in Chapter 3, the survey obtained responses from the lead staff person in each of the 15 
PTC programs and from administrators at college partners in 14 of the 15 program locations. 
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in three-to-five years, and roughly doubling the total number of young adults served in its PTC 
programs over the next five years. 


Doubling was to be achieved through increases in both the number and size of PTC programs. 
Year Up national leaders took steps to see that efforts to increase enrollment did not come at 
the expense of applicant quality. They let local staff know that they should continue to screen 
out young adults who were either not ready for the program or judged to be able to succeed 
without it. National staff regularly monitored local program statistics to make sure the 
characteristics of participants reflected the types of applicants sought. 


There was strong agreement between Year Up and college personnel that PTC programs 
should establish young adults in careers with good potential for upward mobility. In the words of 
several Year Up staff members, the goal is “a W-2 that grows.” This view reflects some 
repositioning of the goal post for Year Up, which historically has focused mostly on shorter-term 
post-program job placements. Year Up national leaders noted that partnering with colleges in 
PTC programs had helped raise awareness of the need and possibilities for doing more to 
encourage continued education and training after participants leave the program. 


Year Up and its college partners had different theories of how best to foster wage growth after 
the program, however. While acknowledging that full-time school might be best for some, the 
dominant view at Year Up was that most graduates will do better in the long-term if they initially 
concentrate on full-time work in Year Up target occupations after graduating from the program. 
Good earnings and job security help to stabilize living situations and provide access to financial 
resources needed to afford additional education and training. 


Training might not mean college: Year Up leaders cited strong possibilities for advancement 
through work-based training and industry certification. In contrast, college partners felt that in 
most cases, continuing in college full-time was the most assured path to credentials needed to 
progress in career pathways. 


The divergence in theories is reflected in the post-program activities local PTC and college 
administrators cited as preferred in the spring 2019 survey. The survey asked respondents to 
specify the percentages of graduates that should be working and going to school immediately 
following program completion in order to foster optimal long-term outcomes. The averages of 
percentages given by PTC program leads were 73 and 34 percent for full-time work and full- 
time school, respectively. The corresponding averages for college partner respondents were 40 
and 55 percent, respectively. 


5.2 Doing as Well with Less 


Year Up designed the PTC program model to reduce Year Up’s costs while generating revenue 
on par with its stand-alone core program. It set the program spending target at around $20,000 
per participant, compared with $35,000 in the core program. Cost reductions were to come from 
lower spending on instruction, rent, and participant stipends. College partners, in theory, would 
recover their costs (for instruction and facilities) by drawing on Pell Grants and other financial 
aid available to low-income students. 


Abt Associates 5 Perspectives on Implementation 1 pg. 59 


To Improve and to Prove 


On the revenue side, generating $20,000 per participant from employer payments for interns 
required local programs to meet two key targets. These targets included retaining at least 83 
percent of enrollees through the completion of L&D, transitioning them to revenue-generating 
internships and collecting Year Up’s full specified payment ($26,000) for at least 90 percent of 
the internships. 


As discussed in Chapter 3, most local PTC programs had not met these targets by the end of 
the study period. Interviews with Year Up national and local leaders provided explored the 
roadblocks encountered. 


One key challenge has been meeting goals for increased recruitment while containing costs. 
Successful recruitment is key to increasing the number of participants and thereby reducing 
average cost per participant. Staff cited varying efforts to respond to this challenge, including: 


e Working with college partners to recruit existing and former students who fit PTC 
program’s target profile. In making the case for strong collaboration on recruitment, 
informants recommended stressing how collaboration could help the colleges achieve their 
goal of serving a more diverse population, address their declining enrollments, and boost 
their completion rates. 


e Exploiting digital advertising and online social networks. Informants felt that better use 
of technology could help to reduce the time staff currently spend in community outreach 
while boosting enrollment. 


e Reducing costs of Year Up’s applicant screening process. Examples here included 
recommendations to strengthen the ability of applicants to self-screen online and generally 
reduce the length and number of steps in the screening process. 


A second significant challenge in reaching financial break-even is raising retention during the 
L&D phase of the program. Here, at least two responses seem promising: 


e Evidence from the mini-experiment (see Chapter 2) suggests that well-focused, low-cost 
academic monitoring and coaching could substantially close the 10 percentage point gap 
between current retention through L&D and the 75 percent retention target. 


e Informants also recommended steps to strengthen the PTC learning communities, by 
adding study halls, increasing the time and variety of activities in advising sessions, offering 
more social and cultural opportunities, and providing more inviting gathering spaces for 
participants and staff. 


Finally, several themes emerged related to challenges in developing internships slots in PTC 
and possible responses: 


e First, PTC programs are relatively new, and it takes time to build relationships with local 
employers in the community. In principle, this challenge should diminish in each local 
program over time. 


e Second, reduced time in learning communities compared to the core program also implies 
less time for participants in the PTC programs to practice and reinforce professional skills 
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employers demand. In addition to the strategies for lengthening learning community time 
mentioned above, recent revisions to Year Up’s professional skills curriculum were designed 
to help strengthen these highly valued professional skills. 


e Third, recent steps to strengthen career planning throughout the program should improve 
the fit between employers’ requirements and interns’ interests and skills. Better aligned 
internships should strengthen employers’ and interns’ experiences and help to stimulate 
increased demand for interns. 


5.3. Meshing Missions 


Year Up national staff, PTC program leads, and the college partners share a strong interest in 
maximizing career development during and after the PTC program. Both parties noted benefits 
from the program partnership in this regard. PTC program leads credited experience with 
colleges as helping them recognize that follow-on education and training are critical to longer- 
term career progress and spurring them to take steps to strengthen career planning to this end. 
College administrators hailed Year Up’s approach to work-based learning and expressed strong 
interest in expanding partnerships with local employers. 


As discussed in Section 5.1, differences in institutional culture and theories of change have led 
to some tension about optimal strategies. For Year Up, the preferred first post-program step is 
full-time employment. For colleges, it is generally full-time school. 


This difference in theories sometimes lead to disagreements about practices. A good example is 
the not infrequent decision about whether to advance participants who did not pass key college 
courses during L&D to internships. Local PTC program staff tended to support such 
advancement, seeing internships as key to their preferred post-program outcome of 
employment. College staff worried that such promotion would deny participants a critical 
opportunity to acquire skills and credits critical for longer-term college success. 


Another practical situation that reportedly has made it challenging to align PTC and college 
missions is the location of some PTC programs at satellite campuses removed from the heart of 
college operations. Local PTC program leads pointed out that such locations make it difficult for 
staff and participants to connect with college personnel and take courses in programs not 
offered at remote locations. Increased exposure to wider campus resources also might 
encourage more participants to continue at the college after they graduate from the PTC 
program. 


More generally, both PTC program staff and college administrators cited a tendency for the PTC 
programs to be insular and not well integrated with other campus activity. One interviewee 
described PTC local programs as sometimes operating more like a “real estate deal” than a 
genuine partnership. 
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Interviewees cited a number of other promising avenues to improved alignment between the 
PTC programs and the colleges: 


e One recommendation was to formalize processes for goals discussions and planning for 
each program at the outset, build decisions into memoranda of agreements, and revisit and 
update agreements at regular intervals thereafter. 


e One college partner described a special orientation course they offered to help PTC 
program participants feel part of the college and familiarize them with college resources. 


e Some colleges have worked closely with PTC program staff on recruitment—variously 
holding joint events and featuring the Year Up’s PTC program in college promotional 
materials. 


e College administrators, advisors, and instructors in some locations regularly participate in 
PTC program activities, and PTC program leads in some offices frequently attend general 
college staff and faculty meetings. 


e Finally, in endorsing Year Up’s recently adopted “guided pathways” approach to career 
planning, one informant urged that PTC programs engage college advisors in an 
authentically collaborative manner. 


As these examples indicate, a number of PTC programs have shown initiative and creativity in 
building bridges. Steps to more fully inventory and disseminate best practices in this arena 
could be beneficial. 


5.4 Recruiting Participants 


In setting recruitment targets for the PTC programs, Year Up seeks to balance its goal of 
increasing enrollment—demonstrating the program’s appeal and scalability—with its 
commitment to maintaining service quality and achieving other target outcomes. In addition to 
showing the program’s appeal to young adults, increased enrollment is key to reducing average 
per participant costs and to helping college partners increase the size and diversity of their 
student bodies. On the other hand, overly ambitious recruitment targets may create pressures to 
admit less suitable young adults who need more support than the programs are prepared to 
provide. With PTC program retention rates already below target levels, Year Up national leaders 
do not wish to encourage growth that cannot be supported. 


As noted in Chapter 3, difficulties with recruitment and retention during the study period led Year 
Up national staff to maintain lower enrollment targets than they had originally envisioned. Most 
local programs still struggled to meet the targets, with the result being slower growth than 
originally envisioned. 


The spring 2019 interviews explored views of the reasons and possible responses to 
recruitment difficulties. Respondents cited broader challenges facing public colleges, as well 
factors specific to the PTC programs. 


Broader challenges included declining numbers of 18- to 24-year-olds nationally and a strong 
economy which improved job opportunities and dampened interest in college. Meanwhile, public 
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colleges have faced steadily increasing competition for students from for-profit schools and 
burgeoning distance learning options, particularly online options. 


Year Up leaders noted that, compared to public colleges, for-profit schools invest substantial 
resources in marketing and recruitment. Overly rosy promises by some reportedly have sown 
deep suspicion within low-income communities that see young adults accumulate debt without 
earning degrees or finding good jobs. PTC program leads report that, as a result, the PTC 
offer—of free college, supports, and an internship—also can seem “too good to be true” at first. 
It often takes repeated contacts over multiple recruitment cycles to build trust and convince 
young adults that the PTC program offer is legitimate. 


Regular college programs also seem to young adults to be an easier route to a career. Such 
programs do not require weekly meetings with coaches and peers, professional skills classes, 
or compliance with behavior contracts. Parents and high school counselors reportedly 
discourage some young adults from enrolling in PTC programs in favor of four-year institutions, 
arguing that the latter provide more valuable degrees and better social opportunities. From Year 
Up’s perspective, the problem with such advice is that many young adults are not ready to 
handle such traditional college programs and the limited supports they have available. 


In cities that operate Year Up’s core program as well as a PTC program, recruiters sometimes 
encounter the opposite problem. Some young adults who have had bad experiences in college 
reportedly prefer the less college-like, more professional environments typical of stand-alone 
programs like Year Up’s core program. 


College partners might seem to be natural allies in recruiting for the PTC program, given the 
shared interest in boosting college enrollments. However, to date, there has not been much 
collaboration on outreach and recruitment. One reason cited is that public community colleges 
typically do not recruit very actively and have limited expertise and resources for recruitment. 
Another reason is that, in wooing college partners, Year Up often has stressed its ability to help 
increase enrollment at little additional cost to colleges. 


Administrative challenges stemming from lengthy college financial aid processes also have 
created recruitment difficulties. Tight college staffing and late aid program determination cycles 
often mean that financial aid decisions are finalized very close to or after the beginning of 
academic terms. Uncertainties prior to that point reportedly lead to significant attrition of 
otherwise well-suited applicants. 


Year Up and college staff identified a variety of promising responses to these recruitment 
challenges, including: 


e Hiring a more diverse recruitment staff, including individuals who have good connections 
with and understanding of the target communities; 


e Building on and formalizing efforts to increase contact frequency and community 
presence across recruitment cycles, through improved use of Year Up alumni, social media, 
and digital marketing campaigns; 
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Striving to maintain applicants’ engagement over the course of the recruitment cycle 
through multiple touch points and interesting activities and through shortening the time 
between application and the start of college classes; 


Strengthening collaboration with college partners, local high schools, and area firms to 
develop more robust applicant pipelines. In addition to tapping existing college applicants, 
informants suggested that colleges might be interested in help from Year Up in mining their 
data on dropouts and encouraging dropouts to re-enroll in college through the PTC 
program.*© One employer suggested engaging corporate partners in outreach to friends and 
family of current employees. 


Considering adjustments to PTC programs that would make them suitable for wider 
populations of disadvantaged young adults. One suggestion was to add a “pre-Year Up” 
program to prepare young adults with lower skills levels and personal and family challenges 
that needed to be resolved. Year Up reportedly also has considered introducing shorter, 
more flexible programs for young adults whose work, family, and financial situations 
preclude a full-time one-year commitment. 


5.5 Improvement Priorities 


In addition to the foregoing specific issues, the interviews also explored stakeholders’ general 
priorities for program improvements. The responses coalesced into three broad categories: 


Ideas for strengthening PTC-college partnerships focused on increased collaboration and 
involvement in each other’s activities. For example, PTC program leads felt it would be 
helpful if college personnel participated in weekly PTC learning community activities such as 
Friday Feedback. One college partner stressed the value of PTC staff and students regularly 
attending college graduation ceremonies and other campus events. 


Informants gave a number of examples of other ways coordination could be improved with 
relatively little effort. To support PTC program recruitment, one respondent suggested that 
college applications solicit information on interest in occupational tracks aligned with the 
PTC programs and another recommended highlighting attractive aspects of college 
campuses in the PTC program marketing materials. A number of local PTC program leads 
described how their programs bridged the gap between college semesters and the 
somewhat longer L&D phase of the PTC program by providing boot camps on useful topics 
not elsewhere covered. 


A second cluster of improvements concerned staffing and other resource needs. PTC 
program leads mentioned admissions staffing as a priority need. College partners noted that 
frequent turnover sometimes makes it difficult to know who to contact and can impede 
forging strong relationships. PTC staff acknowledged that turnover can result in a younger 
and less experienced staff and varied operational challenges. Some PTC staff also identified 
a need for more ample and centrally located space on campus. Finally, some staff felt that 


48 Such help would have to be approached in a way that did not violate applicable data privacy 


restrictions, which have been one source of reluctance to share lists. 
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larger student stipends would help participants to better meet transportation and other 
needs adversely affect participation. 


e The third category included strengthened supports for longer-term career advancement. 
There was strong agreement that continuing skill building and credentialing were crucial. 
Although, as noted earlier in this chapter, views of the optimal balance and sequencing of 
school and work varied. Informants identified a number of steps that could be taken during 
the program to increase college persistence. Generally, these steps involved strengthening 
alignment among interests, courses of study, and work during internships. Related themes 
included a need to maintain academic rigor in PTC courses (emphasized by college 
administrators) and for capacity to quickly revise curricula to address the changing skill 
needs of employers (emphasized by Year Up staff). Chapter 2 discusses related 
recommendations for improved career planning. 


5.6 What Makes the PTC Programs Work? 


When adapting multi-component programs for scaling it is important to have a good handle on 
which services are critical to preserve and at what strength. In designing PTC program model, 
Year Up leaders conducted extensive analysis of core program components to decide which 
were vital to retain in their current form and which could be altered or dropped. The resulting 
PTC program model reflects the organization’s view of the elements that contribute most to 
program effectiveness. 


Components preserved close to their original form included recruitment and screening, 
professional skills training, weekly learning community activities, coaching, and internships. 
Year Up national’s leadership judged these components to be both necessary for a substantial 
impact and difficult to transfer to college partners. In contrast, they reasoned that colleges could 
provide quality instruction in English, math and technical skills. Although Year Up retained 
participant stipends in its PTC program model, it reduced stipend levels, reasoning that many 
participants would receive residual cash to help meet living expenses from Pell and other grants 
(net of tuition). More generally, Year Up’s leaders gambled that PTC programs could achieve 
comparable results to those of core programs with a lower staff-to-participant ratio and fewer 
program contact hours per week. 


The Year Up national staff and PTC program leads we interviewed felt that experience to date 
has validated many of these bets, while leaving others unproven. For example, swapping in 
college instruction for English and technical subjects strengthens participants’ connections to 
longer-term credential programs and affords Year Up access to a wider menu of technical fields 
for training. But relying on college instructors makes it harder for program staff to monitor and 
address academic difficulties encountered by participants. It also has been more difficult than in 
the core programs to revise curricula quickly in response to changing employer needs. More 
generally, both PTC program leads and college administrators acknowledged that aligning 
cultures and practices has been more difficult than envisioned. 


In the views of both Year Up and partner college informants, Year Up’s single most important 
capacity is its ability to engage local employers in providing program participants with valuable 
work-based learning experiences. In the words of one college staff member: “the 
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internship...experience is unique. It provides the experience with what the world is, as opposed 
to anything Year Up can provide in programming or an academic course. | think that’s number 


one. 


Finally, employers interviewed also offered their perspectives on critical Year Up services. They 
emphasized the PTC program’s array of services for instilling strong communication skills and 
work habits and expressed the view that PTC staff were very effective in monitoring and 
troubleshooting internship experiences. 


5.7 Upscaling PTC 


Year Up’s ultimate goal for the PTC programs is to demonstrate the model's effectiveness and 
scalability. Reaching this goal requires showing that programs can operate in multiple locations, 
reach at least a moderate size at each location, and achieve financial break-even (i.e., fully 
cover costs through internship revenue). Year Up does not plan to develop PTC itself on a large 
scale. Rather, the goal is to expand just enough to demonstrate the model’s promise and 
encourage wider dissemination by other organizations. 


Year Up national staff cited the numeric goal of roughly doubling the total number currently 
enrolled annually in the PTC and core programs together—which they put at about 4,700—over 
the next five years, entirely through growth in PTC program. They plan to achieve this growth by 
expanding to additional locations and by increasing annual enrollments to 160 or more at each 
program. As discussed earlier, at 160 participants per year, average costs fall to about $20,000 
per participant. Assuming that retention and internship revenue are on par with core program 
levels, the PTC programs should reach financial break-even at this scale. 


Evidence gathered for this study indicates strong potential for growing to and beyond the break- 
even point. There has been progress on many fronts, and promising responses to other 
challenges are underway. For example: 


e Interviewees reported seeing evidence of strong potential for digital advertising, streamlined 
admissions processes, and effective targeting of college applicant and drop-out pipelines to 
boost recruitment. 


e Evidence from Mini-Study #1 (discussed in Chapter 2) suggests that it should be feasible to 
boost retention at modest cost through improved monitoring of academic performance of 
participants and timely provision of supports to those who encounter academic difficulties. 


e Longer-term efforts to improve Year Up’s capacity to meet large employers’ needs for skilled 
entry-level workers should help create more demand for interns. This so-called “customer 
solutions” approach involves working with large firms to identify important skill gaps in their 
workforces and re-engineer PTC programs to reliably meet this demand. 


In addition to scaling the current PTC program model, Year Up national leaders described other 
ideas for scaling under development. One approach involves further adapting direct service 
models such as PTC and the core program in a manner that preserves basic features but 
permits substantially shifting administrative responsibility to other agencies. According to one 
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Year Up national leader, such a model potentially could reduce Year Up’s direct costs to 
$10,000 per participant and be scalable to 100,000 young adults annually. 


A second idea is to package and widely disseminate PTC services seen as particularly 
efficacious. As an example of this approach, Year Up has developed a package of curricula and 
training materials that would allow other organizations—such as colleges, employers, and 
workforce agencies—to implement its signature professional training course. This curriculum— 
called Career Labs—already has been used in several settings. 


5.8 “To Improve and to Prove” 


In closing, through this and other projects Year Up has demonstrated a strong commitment to 
generating and using evidence in scaling its core and PTC programs. In one leader’s words, this 
approach involves two key steps—“to improve and to prove.” 


The present project—involving collaborative formative studies of the PTC programs— 
exemplifies Year Up’s strong commitment to research-informed improvement. Meanwhile, a 
large, ongoing randomized controlled trial of the core program—which similarly followed years 
of work to perfect implementation and a small pilot randomized controlled triali—shows the value 
of the second step in the improve-prove paradigm. 


This report has documented substantial progress on PTC program implementation to date and 
Year Up’s clear grasp of the remaining challenges. Although the current grant has ended, there 
are still many possibilities for valuable mini-studies to help solve these challenges and a wide 
variety of promising approaches to research collaboration. 


Meanwhile, under a separate grant, the current study team will continue to monitor 
implementation of the PTC programs and measure longer-term impacts from the two small 
randomized controlled trials begun under this project. Once the PTC program model has been 
fully implemented, the team will be working with Year Up to assess potential approaches to a 
large-scale randomized controlled trial to measure its effectiveness. 


Associates 
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Appendix A: Supplemental Information for the Mini-Studies 


Exhibit A-1: Potential Mini-Study Topics Emerging from Stakeholder Outreach 


Recruitment and Admissions 

1. Targeting criteria: What is the ideal participant profile for achieving program outcomes 
and impacts, considering characteristics such as levels of interest/motivation, 
academic ability, and readiness factors, as well as the demographic composition of the 
cohort? How should these criteria be weighted and under what circumstances should 
they vary across time or program sites? 

Pros and cons: Participant recruitment is both a high-cost and highly-influential 
activity with major consequences for later program operations and outcomes. Existing 
data on many key characteristics are missing or inconsistent in quality. 

2. Targeting efficiency: How might Year Up improve the efficiency and effectiveness of 
the outreach and recruitment process, for example, through closer synchronization 
with college partner recruitment pipelines or deepened community partner pipeline 
relationships? In those locations where Year Up Core and PTC sites are co-located, 
are there opportunities for cross-channel coordination? 

Pros and cons: An efficient process for targeting individuals likely to be successful in 
the PTC is essential for creating a pipeline of qualified applicants to support scaling the 
program. A challenge is that potential pipelines could vary greatly across locations, 
making it difficult to develop generalizable findings. 

3. Assessment: Are there more reliable and valid measures of applicants’ basic academic 
skills and psychosocial competencies (e.g., interest and motivation)? How might 
existing measures be improved and what additional assessments might be useful? 
Pros and cons: Improved assessment tools could reduce the risk of enrolling youth 
who are unlikely to succeed with reasonable levels of support services, and they could 
improve the ability of program staff to more quickly and effectively target support 
services to those who are enrolled. A challenge will be reaching the right balance 
between improving on the use of current tools versus acquiring and/or developing and 
implementing better tools. 


Student Supports 

4. Services needs and models of support: What is the profile of support service needs 
across PTC programs and what models of support have programs adopted to address 
these needs? How successful are the various types of partnerships among PTC sites, 
colleges, and community partners in meeting the support service needs of students 
and what types and levels of needs persist? 
Pros and cons: Strong support services are critical to student success, and 
possibilities for sharing responsibilities with colleges both raise challenges and present 
opportunities. A solid assessment of models under development by Year Up staff may 
require more time and resources than available for a mini-study, so it may be 
necessary to make some compromises. The quality of data on service use and 
outcomes is likely to be inconsistent across, and possibly within, sites. 

5. Financial aid: How can PTC programs maximize students’ access to financial aid for 
degree-applicable credits during and after enrollment in Year Up PTC? How can Year 
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Up address developmental education needs/courses for students without requiring 
students to use up their financial aid? 

Pros and cons: The financial viability of the PTC program model depends on the 
ability to cover some program costs through federal and state student financial aid 
programs, particularly grant programs. Addressing this topic as a mini-study will be 
tricky given complexities around what program costs federal and state aid might cover 
and uncertainties about data availability. 

6. Learning communities: What are “best practices” for creating strong learning 
communities in an environment where students in PTC spend substantially less time in 
on-site program activities and otherwise engaging with staff than do students in the 
core program? What are the possibilities for increasing contact hours and cohesion 
through alternative scheduling approaches, changes in use of technology, 
modifications to required activities, and more collaboration between college instructors 
and PTC staff? What qualities of learning environments tend to facilitate and what 
qualities tend to impede success? 

Pros and cons: Strengthening learning communities is a promising avenue for 
achieving outcomes comparable with Year Up’s core program and there may be 
relatively inexpensive ways to do it. A mini-study on this topic will require time from 
college personnel. 


Academics 

7. *Addressing academic difficulties: What are promising approaches to monitoring 
student progress to identify and intervene early in cases of poor/declining course 
performance? What are promising approaches to monitoring performance at the class 
level to identify mismatches between instructional strategies and student outcomes 
and provide constructive feedback to instructors? 

Pros and cons: Working out effective joint approaches to supporting academic 
progress is a key requirement for successful outcomes. Challenges for a study in this 
area include engaging instructors and obtaining data on skills and outcomes needed to 
support a strong analysis. 

8. Aligning skills: Are there ways to improve the alignment between L&D activities and 
skills required for successful internships and post-program placements, for example, 
through workshops, boot camps, and existing courses, or new courses? How can the 
application of input from employers be strengthened? Are there advantages to 
adjusting the balance of time invested in building technical versus basic skills, 
depending upon internship and job needs? 

Pros and cons: There are potentially very large benefits to finding ways to better align 
the skills of students coming out of L&D with those needed for success in specific 
internships, both in terms of employer retention and student success. The issues 
involved are many and complex, however, so a mini-study would need to be carefully 
specified. 

9. Credits and credentials: What strategies might help minimize time devoted to 
remediation in the program? Are there ways to integrate some of the basic skills 
training into the occupational skills training? How can programs maximize degree- 
applicable credits earned? 

Pros and cons: Being able to transform more critical program activities into credit- 
bearing courses would lower the “after tuition cost” of the program and accelerate 
progress of students towards credentials and degrees. A potential weakness of this 
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topic as a mini-study is that related conditions may be specific to local curricula, 
norms, processes, thus making this better suited to a management-consulting task. 


Internships 

10. *Quality of internship experience: What strategies are used at the various PTC sites to 
ensure that all internships provide challenging, career-building work experiences? Are 
there ways they could be developed to be better aligned with career paths 
emphasized in the program? Possibilities include revised approaches to developing 
internships and working with employers. 

Pros and cons: Successful internship experiences are critical for maintaining 
employer support and, thus, for the growth and sustainability of the PTC program 
model. Improving and applying findings about what factors are most critical for quality 
internship experiences could be very valuable. The potential for applying lessons to 
improve outcomes is uncertain given the number and variety of company relationships 
involved. 

11. Challenging internship sales markets: What is the range of strategies used in 
internship sales and what are defining qualities and contexts for the more and less 
successful efforts? What improvements could be made to marketing and sales 
approaches to respond to challenges in developing internship sales in different local 
areas? 

Pros and cons: Strong internship sales are crucial to the program’s financial viability. 
It may be a good time to assess initial experience and best practices following 
“customer solutions” approaches. On the other hand, in a short-duration study, it may 
be difficult to learn more about practices and issues in local markets than Year Up 
staff already know. 


Alumni Services 

12. Capturing post-program career trajectories: How might Year Up strengthen and extend 
data collection and analysis of post-program data on employment and education 
outcomes? 

Pros and cons: Solid data and analysis of longer-term outcomes is critical to 
understanding Year Up’s role in employment and post-secondary education and 
navigating any tensions therein. Although there may be important implications for 
program goals and strategies, this mini-study would not be as directly focused on 
program improvements as other topics. 

13. *Employers’ role in ongoing education: What are some of the promising approaches 
employers are currently using or might adopt to support graduates’ college completion 
after PTC? How can Year Up increase employers’ commitment to support in this area? 
Pros and cons: A study in this area could create pathways allowing interns to earn 
credits for their work experience during and/or after their internships, thereby 
accelerating time to certificate and degree completion. The main challenge is that it 
may be difficult to identify strategies for expanding post-program employer supports 
for continuing education. 


Source: Internal program communication, Institute of Education Sciences (IES) Evaluation: Potential Mini-Study 
Topics Emerging from Stakeholder Outreach, March 2016. 


* Denotes topics that were further developed in one of the three mini-studies described in Chapter 2. 


Abt Associates Appendix | pg. 70 


To Improve and to Prove 


Exhibit A-2: Mini-Study #1 — Sample Sizes, by Study Group, Location, and Testing Cycle 
Total Enrolled 


Improvement LUTTE. Improvement UESTE] Improvement UESTE] 
Strategies Strategies Strategies Strategies Strategies Strategies 
Moor} 4(0) 1) Group Group Total Group Group Total Group Group Total 
Site 1 27 27 54 26 30 56 53 57 110 
Site 2 19 20 39 18 22 40 37 42 19 
Site 3 30 28 58 34 36 70 64 64 128 
Total 76 75 151 78 88 166 154 163 317 


Source: Year Up Salesforce® data. 


Note: Accepted applicants who attended orientation were randomized to the Improvement Strategies Group or to the 
Usual Strategies Group. No participants switched condition during the study. 
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Exhibit A-3: Mini-Study #1 - Demographic and Background Characteristics of the Study Sample, 


by Study Group 
Study Group 
Total Improvement USTE)| 
Sample Strategies Group Strategies Group Difference p-Value 
Gender = Female 49.5 48.7 50.3 -16 1751 
Race-Ethnicity 2435 
Black or African American 71.6 14 18 -0.4 
Hispanic or Latino 15.2 12.3 17.2 -49 
White or Other Race 14.0 16.2 11.0 5.2 
Age .1620 
Under 20 44.4 47.4 41.1 6.3 
20-22 41.1 35.1 45.4 -10.3 
23 or older 15.7 17.5 13.5 4.0 
Number of Risk Factors 8740 
0-1 26.3 24.1 28.3 -4.2 
2-3 36.2 36.9 35.5 14 
4-5 16.9 17.0 16.7 0.3 
6+ 20.9 22.0 19.6 24 
Prior College (Any) 4903 
0 years 41.4 39.6 42.9 -3.3 
<1 years 26.9 25.3 28.2 -2.9 
1+ years 32.2 35.1 28.8 6.3 
Prior College (FTE) 1962 
0 years 41.4 39.6 42.9 -3.3 
<1 years 37.2 37.7 36.8 0.9 
1+ years 21.5 22.7 20.2 2.5 
Sample Size 317 154 163 317 


Source: Year Up administrative data from program applications. 


Notes: The study sample consists of opportunity youth who were first enrolled in Year Up with the January 2018 or 
July 2018 cohort at the three sites. Data on number of risk factors is missing for 12 percent of the study sample. 


The statistical significance levels of differences across sample subgroups are denoted by asterisks: * denotes less 
than 10, ** less than 5, and *** less than 1 level on a two-tailed test. 
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Exhibit A-4: Mini-Study #1 — Usual Strategies and Improvement Strategies Tested, by Testing Cycle and Site 


Cycle 1 


Strategy Site1 Site2 Site3 Site1  Site2  Site3 
Communication and information sharing between YU and college/instructors 

Usual Strategy 

All instructors invited to meet with YU Staff for Orientation; staff conduct periodic classroom observations to monitor attendance, professional dress, etc. 

Improvement Strategies 


--Initial Plan 

Develop & implement prescriptive monitoring protocol to track assignment completion & academic JV JV JV J J J 
performance indicators; tailor outreach to instructors at beginning of term; host instructor luncheons 

--lmplementation 

Created shared spreadsheet containing feedback from instructors; share of grades by some instructors if A JV JV 

Tailored outreach to instructors at beginning of term JV JV JV 

Host instructor luncheon JV JV 
--Modifications 

Incorporate information from coaching sessions into shared spreadsheets; lower expectation of collecting JV nA JV 


grades from instructors 


Systems & routines for managing participants' academic information 
Usual Strategy 


Limited sharing of information from Year Up administrative data (i.¢., information related to the Year Up contract like class attendance & professional dress); no 
intentional discussion of academic issues in learning community meetings or establishment of academic support plans 


Improvement Strategies 


--Initial Plan 

Site leadership shares academic history information from admissions; coaches & academic coordinator JV JV JV 

share information from prescriptive monitoring of academics 

--lmplementation 

Site managers routinely reach out to instructors to collect academic information & share it with coaches JV Vv 


of participants in Improvement Strategies Group 


Abt Associates Appendix | pg. 73 


To Improve and to Prove 


Cycle 1 
Strategy Site1 Site2  Site3 
Academic Coordinator designated to reach out to instructors & others at the college; shares info with Vv 
coaches of participants in Improvement Strategies Group; flags "students of note" in a spreadsheet 
--Modifications 


Academic issues surfaced in coaching are discussed in LC meetings; participants of concern are flagged 
& academic support plans put in place 


Attention to Academics During Coaching 
Usual Strategies. 


Site 1 


Site2  Site3 


All coaches provide coaching as usual with focus on YU Topic of the week and professional skill development; participants receive introduction and orientation to L&D 


portfolio project (no academic focus) 
Improvement Strategies 
--Initial plan 


Increased focus on academics during coaching; participants introduced to expanded portfolio assignment JV JV JV 
that includes focus on academics (coaches to check in and advise) 


--lmplementation 

Increased focus on academics during coaching; informal flagging of participants' academic issues 
Use of one-page guide to facilitate academically-focused conversations in coaching 

Participants pull up grades from college LMS during coaching 

Expanded portfolio assignment focusing on academics (coaches check in & advise participants) 
--Modifications 


Ka N\A 4K 
i 


Coach training on Weekly Academic Coaching Notes Sheet & The Binder; coaches routinely flag 
students of concern & have participants pull up grades from college LMS; lower expectation of including 
academic component in portfolio assignment 

Additional Supports 


Usual Strategies 

Participants encouraged to seek out tutoring should they need it; remind participants of online tutoring 

options 

Improvement Strategies 

--Initial plan 

Increase referrals to existing college tutoring & support services, as well as online tutoring JV Vv Vv 
Create textbook library Jf 


Abt Associates 
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Cycle 1 Cycle 2 
Strategy “Site1 Site2 Site3 Site1 Site2 Site3 
Hire Academic Coordinator to collect data on participants' academic performance & engagement from 
college staff to share with coaches Vv JV 
Order Wi-Fi Hotspots for participants in Improvement Strategies Group to use Z V 
--lmplementation 
Increased referrals to existing college and on-line tutoring & support services Vv Vv Jv JV JV JV 
Created textbook library JV Vv 
Hired Academic Coordinator JV JV 
--Modifications 
Provide additional coaching to encourage use available academic and non-academic supports & f ¥ JV 


develop a compendium of available supports 
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Exhibit A-5: Mini-Study #1 — Characteristics of Sample Members Randomly Assigned to 
Improvement and Usual Strategies Groups 


Study Group 
Total Improvement UTE) 
Sample Strategies Group Strategies Group Difference p-Value 
Gender = Female 49.5 48.7 50.3 -1.6 0.7751 
Race-ethnicity 0.2435 
Black or African American 71.6 14 18 -0.4 
Hispanic or Latino 15.2 12.3 17.2 49 
White or Another Race 14.0 16.2 11.0 5.2 
Age 0.1620 
Under 20 44.4 47.4 41.1 6.3 
20-22 41.1 35.1 45.4 -10.3 
23 or older 15.7 17.5 13.5 4.0 
Number of Risk Factors 0.8740 
0-1 26.3 24.1 28.3 4.2 
2-3 36.2 36.9 35.5 14 
4-5 16.9 17.0 16.7 0.3 
6+ 20.9 22.0 19.6 24 
Prior College (Any) 0.4903 
0 years 41.4 39.6 42.9 -3.3 
<1 year 26.9 25.3 28.2 -2.9 
1+ years 32.2 35.1 28.8 6.3 
Prior College (FTE) 0.7962 
0 years 41.4 39.6 42.9 -3.3 
<1 year 37.2 37.7 36.8 0.9 
1+ years 21.5 22.7 20.2 2.5 
Sample Size 317 154 163 317 


Source: Data are from the Year Up administrative data system. 


Notes: These data were weighted to account for the blocking of participants prior to randomization. The estimates of 
the mean difference between the treatment groups are based on regression models that included covariates for 
participant characteristics at the time of program enrollment. Means for the Usual Strategies Group are unadjusted 
means. 

a p-Values for tests of differences in impacts by enrollment cohort/cycle and by office are listed above those for the 
individual impact estimates. 


* = statistically significant at the .10 level, ** =.05 level, and *** =.001 level on two-tailed tests. 
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Exhibit A-6: Estimated Impacts of the Improvement Strategies for Academic Monitoring and 
Supports on College Enrollment in the Month Following the Scheduled Completion 


of L&D 
Group Means (%) Difference in Means 95% Confidence Interval 
Improvement UTE} Lower 
Strategies Strategies %-Point p-Value 2 Bound Upper Bound 

Total Sample 67.2 54.0 13:3. 0.003 6.0 20.5 
Testing Cycle 0.089 

1 January 2018 47.2 41.3 5.9 0.334 41 15.8 

2 July 2018 84.8 64.8 20.0 * 0.002 9.5 30.5 
Program 0.515 

Site 1 42.0 26.3 15.7 ™ 0.031 3.8 276 

Site 2 99.6 92.9 6.7 0.196 -1.8 15.3 

Site 3 68.4 53.1 15.3 * 0.076 1.2 29.4 
Sample Size 154 163 317 


Source: Data on college enrollment are from the National Student Clearinghouse. 


Notes: These data were weighted to account for the blocking of participants prior to randomization. The estimates of 
the mean difference between the treatment groups are based on regression models that included covariates for 
participant characteristics at the time of program enrollment. Means for the Usual Strategies Group are unadjusted 
means. 

4 p-Values for tests of differences in impacts by enrollment cohort/cycle and by office are listed above those for the 
individual impact estimates. 


* = statistically significant at the .10 level, ** =.05 level, and *** =.001 level on two-tailed tests. 
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Exhibit A-7: The Academic Coaching Binder 


Throughout cycle 1 of Mini-Study #1 (see Chapter 2), three study sites implemented staff- 
developed improvement strategies intended to strengthen academic performance during 
and retention through the Learning and Development (L&D) phase of the program with a 
random subset of participants, termed the Improvement Strategies Group. A large portion 
of the improvements were associated how staff approached their roles as coaches for 
participants. Program staff followed their usual strategies for monitoring and supporting 
participants not assigned to the improvement strategies group—the Usual Strategies 
Group. 


Based on experiences with the improvement strategies in cycle 1, program staff at each 
study site modified the improvement strategies they used with the Improvement Strategies 
Group but continued to follow their usual strategies with other participants. Staff in each 
study site used feedback from coaches, program leads, and college partners in cycle 1 to 
modify the improvement strategies during cycle 2 of testing. The common element in the 
improvements between cycles 1 and 2 was the adoption of a tool—The Academic 
Coaching Binder—that Year Up national evaluation staff and the study team assembled 
based on materials program staff had pulled from existing Year Up resources or created 
“on the fly” to support their improvement efforts (see Baelen et al., 2020). 


The Binder was designed as a means of sharing some of the lessons and tools from 
cycle 1 of the study to help coaches in cycle 2 meet expectations for better monitoring of 
participants’ academic performance and more efficiently support participants found to be 
encountering difficulties. While most tools in the Binder were exact versions of existing 
tools or slight adaptations (e.g., infusing more of an academic focus, some were newly 
created based on insights from the experiences of coaches working with the Improvement 
Strategies Group during cycle 1 of testing. 


The primary tool for collecting participant academic information is the Weekly Academic 
Coaching Notes Sheet, which was an updated version of the one-page coaching guide 
used created by staff for use in cycle 1. With the Notes Sheet, coaches working with 
participants in the Improvement Strategies Group could record critical academic 
performance and related information during weekly one-on-one sessions (p. 8). This 
information was most often collected by participants pulling up their grades and course 
information (e.g., assignments, syllabi) in their college learning management system 
(LMS). Participants were also encouraged to fill out a “Preparation Sheet” (p. 5-7) prior to 
coaching to promote greater awareness and reflection about academic issues. 


Coaches routinely shared their Notes Sheets with their Program Manager and their peer 
coaches during learning community (LC) meetings. Use and sharing of the Notes Sheet 
led to more frequent discussion of grades during one-on-one and group coaching 
sessions, as well as enhanced communication among program staff during LC meetings 
about participants’ academic performance and support needs. 
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In addition to the Notes Sheet, the Binder includes a compendium of resources designed to 
help coaches working with the Improvement Strategies Group during cycle 2 of testing better 
monitor academic performance and more quickly and effectively identify causes of and 
respond to lapses in performance. Examples of these other resources in the binder are tips 
for engaging with instructors, tips for troubleshooting academic challenges and goal setting 
templates. The Binder also includes several time-management resources (inserted by the 
study team) In feedback from coaches during cycle 1 noting that time management was an 
area in which they often struggled. 


The Binder, particularly the Notes Sheet, reportedly were instrumental in equipping coaches 
working with participants in the Improvement Strategies Group to establish routines in 
coaching around academics and avail them of tools for soliciting information from 
participants about their academics and providing needed supports to participants facing 
academic difficulties. Subsequent to the completion of mini-study 1, which found evidence of 
substantial improvements in program retention through L&D among those in the 
Improvement Strategies Group, Year Up has made the Binder available to all PTC program 
staff and trained coaches in its use. 
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Appendix B: Supplemental Information for the Impact Evaluation 


Exhibit B-1: Sites Selected for the Impact Evaluation 


Study Site 


Characteristic Site 2 SIGIR All PTC Programs 

Background Characteristics 

Dual-channel- Core and PTC Programs No Yes Yes NA 

Date First Cohort Enrolled Summer 2010 Summer 2015 Winter 2016 

Cohorts Enrolled in the Impact Evaluation NA 
Summer 2017 v 
Winter 2018 v v 
Summer 2018 v v 
Winter 2019 ‘ v 

Program Performance Measures 

% Completing Learning & Development 778 775 79.0 75.6 
(L&D) Phase (Target = is 83%) 

% of L&D Completers Graduating from the 92.9 76.0 93.8 84.1 
PTC Program (Target = 90%) 

% of Enrollees Graduating from the PTC 72.3 58.9 74.1 63.6 
Program 

% of Program Graduates Entering Full- 55.1 49.3 63.3 59.1 


time Employment in Target Occupation 
(Target = 70%) 


Source: Year Up administrative data. 


Note: Performance measures are averaged for the two cohorts enrolling prior to the date that each site began intake 
for the experiment. 
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Exhibit B-2: Characteristics of Youth Enrolled in PTC Programs, Calendar Year 2016 


AIlPTC Non-Study PTC PTC Programs in the Impact Evaluation 


Programs Programs Site 1 Site 2 Site3 _‘ Total 


Participant Characteristics 


Gender-Male (%) 56.6 56.8 55.3 64.2 44.4 55.9 
Age 20 or Older (%) 55.5 53.3 64.7 67.5 44.4 60.1 
Race Ethnicity (%) 

Black/African American 48.4 45.7 84.7 50.8 27.2 54.2 

Hispanic or Latino 30.2 35.6 24 9.2 49.4 18.5 

White or Other 21.4 18.7 12.9 40.0 23.5 27.3 
Prior College Enrollment (%) 47.3 46.1 35.3 60.8 49.4 50.0 
Risk Factors (%) 

0-1 32.4 28.2 69.4 31.7 27.2 41.6 

2-3 27.8 27.7 17.6 25.8 42.0 28.0 

5 or More or Not Recorded 39.8 44.1 12.9 42.5 30.9 30.4 
Program Performance Metrics 

Completed L&D (%) 79.3 78.7 76.5 775 88.9 80.4 

Graduated (%) 67.4 66.5 70.6 57.5 85.2 69.2 
Number of Youth 907 621 85 120 81 286 


Source: Year Up administrative data. 
Note: See description of the site selection criteria and the characteristics of the selected sites in Appendix Exhibit B-1. 
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Exhibit B-3: Summary of Sample Enrollment for the Impact Evaluation 


Enrollment Scheduled Number Assigned Randomization Period 
Cohort & Program Treatment Control _ % 
Block Completion Group Group _‘ Treatment Earliest Latest 
Site 1 1.41 Jul-2018 94 45 66.7 4/4/2017 8/21/2017 
ee Jul-2018 2 2 50.0 7/17/2017 7/17/2017 
2.1 Jan. 2019 80 40 66.7 9/22/2017 1/9/2018 
Site 2 2.1 Jan. 2019 53 14 778 10/26/2017 3/9/2018 
3.1 Jul-2019 23 8 72.7 5/16/2018 7/25/2018 
4.1 Jan. 2020 38 11 778 10/24/2018 2/27/2019 
4.2 Jan. 2020 12 9 57.1 3/6/2019 3/26/2019 
Site 3 3.1 Jul-2019 14 6 66.7 5/4/2018 5/18/2018 
3.2 Jul-2019 7 2 80.0 7/6/2018 7/20/2018 
3.3 Jul-2019 9 2 82.9 6/1/2018 7/20/2018 
3.4 Jul-2019 20 4 83.3 6/1/2018 6/29/2018 
4.1 Jan. 2020 31 14 66.7 10/22/2018 12/28/2018 
4.2 Jan. 2020 6 6 50.0 1/11/2019 1/17/2019 
Total 389 163 70.5 7/17/2017 3/26/2019 


Source: Year Up’s administrative data. 
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Exhibit B-4: Recruitment and Randomization 


In designing the PTC impact evaluation, the research team sought to create a large 
enough study sample to be able to detect meaningful impacts without putting undue 
burden on program recruitment operations. Under ideal conditions it would be possible to 
set a ratio of treatment to control groups between 1:1 and 2:1. In practice, that proved to 
be challenging due to recruitment difficulties. 


In response, the team adjusted the random assignment ratios several times in order to 
balance program needs to meet their enrollment targets versus the need for an adequate 
research sample. Throughout the four enrollment periods, the team used eight different 
probabilities of assignment to the treatment group, ranging from 50 percent to 83 percent 
(see Appendix Exhibit B-3). 


In some offices it was necessary to suspend random assignment for a period. During 
those periods, programs were allowed to admit all new applicants they identified. Over the 
study period, a total of 166 program-eligible applicants were identified during such periods; 
they are not included in the study sample. 
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Exhibit B-5: Data for the Impact Evaluation 


Data Sources 


There were three main sources of data for the study: Year Up’s administrative data, the 
National Student Clearinghouse (NSC) and the National Directory of New Hires (NDNH). 


Year Up’s administrative database. Year Up tracks its applicants and enrollees ina 
Salesforce® customer-relationship management database. Year Up shared with the study 
team a set of files from this database, created on October 7, 2019, and describing 
applicants’ baseline characteristics (Such as age, gender, and race-ethnicity), as well as 
admissions and program outcomes (such as enrollment and completion of the program). 
These data are available for the full study sample. 


National Student Clearinghouse (NSC). The NSC is a private organization that collects 
student enrollment and credential records from participating colleges across the nation. 
College participation rates are high, and according to the Clearinghouse’s research arm, 
the NSC’s enrollment records in the fall of 2017 represented 97 percent of all enrollments 
reported to the U.S. Department of Education’s Integrated Postsecondary Education Data 
System (IPEDS) at Title IV, degree-granting institutions. Coverage is high among both 
public and private not-for-profit colleges (99% and 95% of enrollments, respectively), but 
lower among private for-profit colleges; 71 percent of the 1.1 million students enrolled in 
private for-profit colleges in fall 2017 were covered in the NSC’s records. In August 2019, 
Year Up obtained enrollment and credential records from the NSC and, with NSC’s 
agreement, shared those data with the study team. 


These data are available through June 2019. We have NSC data for this study covering 
quarters 1 and 2 post enrollment for all of the study sample; data for quarters 1 through 4 
for the first three enrollment cohorts; and data for quarters 1 through 5 for the first two 
enrollment cohorts. 


National Directory of New Hires (NDNH). Data on quarterly earnings were drawn from 
the NDNH, a national database operated by the Office of Child Support Enforcement 
(OCSE) in the U.S. Department of Health & Human Services. The NDNH collects quarterly 
wage data on individual employees from state workforce agencies and federal agencies. 
Generally, these data do not include earnings from self-employment, but some states do 
require reporting for independent contractors (though none of the three states in which the 
RCT offices were located required independent contractor reporting). Naturally, the NDNH 
also does not have data on earnings from unreported, “off-the-books” employment, nor 
does the NDNH include the stipend PTC participants earn from Year Up. 


These data are available through June 2019. As for NSC, the data cover quarters 1 and 2 

post enrollment for all of the study sample, quarters 1 through 4 for the first three 

enrollment cohorts, and quarters 1 through 5 for the first two enrollment cohorts. 
(continued) 
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Alignment of Data with Phases of Program Activity. For analytic purpose we map the 
outcomes data to reference points in program time. Quarter 0 is defined as the calendar 
quarter in which, if assigned to the Treatment Group, sample members would have been 
invited to enroll in the PTC program. Programs enrolled new cohorts roughly every six 
months. If the program began in August 2017, Quarter 0 would cover July through 
September of 2017; Quarter 1 would cover October through December of 2017, and each 
successive quarter would cover the next three months. 


Site 1 enrolled the first cohort of students in the study sample in August of 2017. Thus, 
Quarter 0 (the enrollment quarter) for cohort 1 is the period July thru September 2017. 
Quarters 1 and 2 (October through December and January through March, respectively) 
roughly align with the time Treatment group members’ enrollment in Quarter 0 would have 
been in the Learning and Development phase of the PTC program and Quarters 3 and 4 
roughly align with time they would have been in their internships. Quarters 5 and beyond 
are post-program. 


Both sites 1 and 2 enrolled youth in second cohort of for study sample. Treatment group 
members enrolled as part of this recruitment cycle began participating in the PTC 
programs in January and March, respectively. Thus, the Quarter 0 for this group is 
January through March 2018, Quarter 1 is the period April through June, and so forth. 


Sites 2 and 3 contributed sample for a third cohort. The program period for cohort 3 
sample members assigned to the treatment group began in September and August 2018 
for individuals in sites 2 and 3, respectively. Then, sites 2 and 3 contributed sample for a 
fourth cohort in early 2019. For those in Site 2, program activities for the treatment group 
began in March 2019 and for those in site 3, program activities for the treatment group 
began in January 2019. 


Appendix Exhibit B-6 provides a breakdown of the study sample used for analysis of each 
of the four outcome measures. This shows the enrollment cohorts and sites with data for 
periods corresponding to different phases of the program, the sizes of the treatment and 
control groups with data for those outcomes and time periods, and the corresponding 
treatment assignment probability. Appendix Exhibit B-7 details the time frame for program 
and college activities for the four cohorts of youth enrolled in the study sample. 
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Exhibit B-6: Sample Enrollment for the Impact Evaluation, by Study Group and Duration of 


Follow-Up 
Number Assigned to: 
Treatment Control _‘Total % Assigned 
Outcome & Follow-up Quarter 00) Toa) Sites Group Group Sample to Treatment 
College Enrollment Outcomes 
Quarters 1 & 2: L&D (All or Some) 1-3 All 302 123 425 71.1 
Quarter 3: Internship (All) 1-3 All 302 123 425 71.1 
Quarters 4 Internship (All or Some) 1-2 #1,2 229 101 330 69.4 
Quarter 5: Post-Program 1-2 #1,2 229 101 330 69.4 
Employment Outcomes 
Quarters 1 & 2: L&D (All or Some) 1-3 All 292 119 411 71.0 
Quarter 3: Internship (All) 1-3 All 292 119 411 71.0 
Quarter 4: Internship (All or Some) 1-2 #1,2 221 97 318 69.4 
Quarter 5: Post-Program 1-2 #1,2 221 97 318 69.4 


Source: Year Up administrative data. 


Note: “All” and “Some” designations in column 1 identify whether the specified follow-up quarters fell within the 
indicated program phase completely or included some months that preceded or followed the indicated program 
phase. 
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R ee 
Sample PTC zt i RCT Quarter 0° RCT Quarter 1 RCT Quarter 2 RCT Quarter 3 RCT Quarter 4 
Cohort _ Site L&D Began M1 M2 M3 | M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 
8/2017 L&D Internship 
1 1 
College Term— | Summer Fall -Required Spring -Optional Fall- Optional 
, 1/2018 L&D Internship 
College Term— Spring- Required Summer Fall- Optional 
2 
4 3/2018 L&D Internship 
College Term— Winter Spring- Required Summer Fall- Optional |__Winter- Optional | 
: 9/2018 L&D Internship 
College Term— Summer Fall-Required |__Winter- Optional _| Spring- Optional Summer-Optional 
3 
8/2018 L&D Internship 
3 
College Term— | Summer Fall-Required Spring- Optional Fall- Optional 
; 3/2019 L&D Internship 
College Term— Winter Spring- Required Summer Fall- Optional 
4 
1/2019 L&D Internship 
College Term— Spring Term- Required Summer Fall Term-Optional | 


* RCT follow-up quarters correspond to calendar quarters indexed in terms of the number of quarters elapsed since random assignment. That is, Quarter 0 is the 


calendar quarter of random assignment, Quarter 1 is the following quarter, and so on. Note that impact tables in this report exclude Quarter 0 and show results 
only from Quarter 1 on to reflect quarters in which the full sample had been randomly assigned. 
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Exhibit B-8: Analysis Approach for Estimating Program Impacts 


Random assignment creates treatment and control groups that will differ in their 
background characteristics only due to chance. With reasonable size samples, low to no 
sample attrition, consistent assignment probabilities for the entire study sample, and 
similarly reliable measures for the treatment and control groups, the two groups should 
have similar characteristics at baseline. All measured differences in baseline 
characteristics would be due to chance and unbiased estimates of program impacts could 
be generated by comparing treatment and control group means for the outcomes of 
interest. 


For this study, we varied the probabilities of assignment to the treatment and control 
groups over time and across sites. This means that, in order to generate unbiased 
estimates of impacts, we need to weight the sample observations by the inverse of their 
treatment assignment probabilities in order to come up with unbiased estimates of both 
baseline differences in sample characteristics and the outcomes of interest. For instance, 
a treatment group member with a two-thirds probability of assignment to the treatment 
group would receive a weight of 1.5 (or three divided by two). Conversely, a control group 
member who had a one-third probability of having been assigned that status would 
receive a weight of 3 (three divided by one). 


To improve the precision of these estimates and adjust for differences between the two 
groups that arose despite randomization, the study team followed the common practice of 
fitting a regression model. For each outcome reported in this chapter, the study team 
estimated the following linear model, using the weighted least squares (OLS) method. The 
basic form of the models can be expressed as: 


Y=X f+or, +e, 
where 
Y, = the outcome for the /* participant 
T, = dichotomous variable indicating assignment to the treatment group 
X ;= vector of baseline covariates 
7B = vector of parameters describing the estimated marginal effect of a one-unit 


change in the corresponding covariate on the outcome measure 


O =estimated effect of having been assigned to the treatment group on the 
outcome 


¢, = error term, which is assumed to have a mean of zero 


The models for all outcome measures contained the following common set of covariates: 


PTC program (binary indicators) 
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Cohort (binary indicators) 


Occupational training track (binary indicator for in a technology or software 
development track) 


Gender (binary indicator) 
Race-ethnicity (binary indicators) 


Risk score (continuous measure; measure squared; and binary indicator of “no risk 
factor’) 


Factors of success score (continuous measure; average rating on four factors: critical 
thinking; interpersonal awareness; persistence; and striving to learn) 


Missing data indicators (binary indicators for each covariate indicating whether the 
information was missing) 


Models estimating college enrollment outcomes included the following additional covariates:! 
Years full-time-equivalent college at enrollment (continuous measure) 
College credential at enrollment (binary indicator) 


Models on earnings outcomes included the following additional covariates: ' 
Earnings in the quarter prior to enrolling in PTC (continuous measure) 


Any employment in the quarter prior to enrolling in the PTC program (binary indicator) 
When the study sample included individuals enrolled over multiple cohorts, the models also 
included variables that were the interactions of the Treatment Group indicator and the Cohort 


Indicator. 


All models included weights denoting the inverse of the probability an individual sample member 
ended up in the treatment status assigned to him/her. 
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Exhibit B-9: Estimated Treatment on the Treated (TOT) Impacts for the Program Period 


tc) (4) Estimated Mean TOT Impact 
Treatment Control €)) C) 


Outcome & Group Group TOT Impact? Standard (5) 
Program Period Mean? Mean> (Col. 1 - Col. 2) Error p-Valuee 
College Enrollment (Cohorts 1-3 for Quarters 1-3; Cohorts 1-2 for Quarter 4) 

Any College Enrollment (%) 


Quarter 1 775 37.7 39.7 *** 46 <.0001 
Quarter 2 92.7 37.0 55.7 =(*** 48 <.0001 
Quarter 3 69.7 33.9 35.7 ** 49 <.0001 
Quarter 4 21.3 23.0 ar. 4.8 0032 
Cumulative Months of Full-time Enrollment 
Quarter 1 44 1.3 28 ai 0.2 <.0001 
Quarter 2 5.2 2.1 3.1 sii 0.3 <.0001 
Quarter 3 5.8 27 3.0 of 0.3 <.0001 
Quarter 4 5.7 2.2 3.6 ae 0.4 <.0001 
Employment & Earnings (Cohorts 1-3 for Quarters 1-3; Cohorts 1-2 for Quarter 4) 
Any Employment (%) 
Quarter 1 53.1 67.4 143 * 49 0312 
Quarter 2 43.9 77.2 -33.3*** 45 <.0001 
Quarter 3 47.2 74.2 -27.0 * 49 <.0001 
Quarter 4 63.8 73.2 -9.4 53 1756 
Total Earnings ($) 
Quarter 1 1,182 3,120 1,938 334 <.0001 
Quarter 2 942 3,135 -2,193 *** 342 <.0001 
Quarter 3 1,128 3,571 2,443 359 <.0001 
Quarter 4 2,115 3,158 1,043 * 436 0112 
Sample Size (Cohorts 1-3/Cohorts 1-2) 
College Outcomes 302/229 123/101 425/330 
Employment Outcomes 292/221 119/97 411/318 


Source: College enrollment information from the National Student Clearinghouse. Employment and earnings data 
from the National Directory of New Hires database. 


Note: These estimates are adjusted to reflect the fact that not all of the treatment group participated in the program. 
The estimation methods are described in Exhibit B 4-8. 


* = statistically significant at the .10 level, ** =.05 level, and *** =.001 level on two-tailed tests. 
4 Treatment group means and the estimated mean impacts of the program are regression adjusted. 


> Control group means are not regression adjusted. They are weighted to reflect to account for differential 
probabilities of assignment to the treatment condition. 
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Exhibit B-10: Treatment on the Treated (TOT) Impacts for the First Post-Program Quarter 


Estimated Mean TOT Impact 
(1) 4) (3) 


Treatment Control Group Estimated Mean Standard 
Outcome & Program Period Group Mean? Mean> TOT Impact Error p-Valuee 


College Enrollment Outcomes (Cohorts 1 & 2) 
Any College Enrollment (%) 


Quarter 5 18.1 23.0 49 5.2 0.4631 
Cumulative Full-time Months Enrolled in College 

Quarter 5 6.1 2.6 3.5 0.4 <.0001 
Employment & Earnings Outcomes (Cohorts 1 & 2) 
Any Employment (%) 
Quarter 5 776 78.2 -0.6 49 0.9252 
Total Earnings ($) 
Quarter 5 4,799 4,292 507 470 0.4138 
Sample Size 

College Outcomes 229 101 330 

Employment Outcomes 221 97 318 


Source: College enrollment information is from the National Student Clearinghouse and employment and earnings 
data are from the National Registry of New Hires (NDNH) database. 


Note: These estimates are adjusted to reflect the fact that not all of the treatment group participated in the program. 
The estimation methods are described in Exhibit B 4-8. 


* = statistically significant at the .10 level, ** =.05 level, and *** =.001 level on two-tailed tests. 
4 Treatment group means and the estimated mean impacts of the program are regression-adjusted. 


> Control group means are not regression adjusted. They are weighted to reflect to account for differential 
probabilities of assignment to the treatment condition. 


Abt Associates Appendix | pg. 91 


To Improve and to Prove 


References 


Alfeld, C., Charner, |., Johnson, L., & Watts, E. (2013). Work-based learning opportunities for 
high school students. Louisville, KY: National Research Center for Career and Technical 
Education. 


Anderson, J. C., Rungtusanatham, M., Schroeder, R. G., & Devaraj, S. (1995). A path analytic 
model of a theory of quality management underlying the Deming management method: 
Preliminary empirical findings. Decision Sciences, 26(5), 637-658. 


Attewell, P., & Monaghan, D. (2016). How many credits should an undergraduate take? 
Research on Higher Education, 57, 682-713. 


Baelen, R., Britt, J., Maynard, R., Souvanna, P. & Warfield, G. (2020). To Improve and to prove: 
Tools to improve academic monitoring and support for young adults. Rockville, MD: Abt 
Associates Inc. 


Belfield, C., & Bailey, T. (2017). The labor market returns to sub-baccalaureate college: A 
review. New York: Center for Analysis of Postsecondary Education and Employment. 


Belfield, C. R., Levin, H. M., & Rosen, R. (2012). The economic value of opportunity youth. New 
York: Corporation for National and Community Service. 


Binder, A. J., & Bound, J. (2019). The declining labor market prospects of less-educated men. 
Journal of Economic Perspectives, 33(2), 162-190. 


Bishop, J. B., & Brenneman, K. A. (1986). An initial assessment of a counseling center’s role in 
retention. Journal of College Student Personnel, 27, 461-462. 


Bloom, H. S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation 
Review, 8(2), 225-246. 


Bound, J., Lovenheim, M. F., & Turner, S. (2012). Increasing time to baccalaureate degree in 
the United States. Education Finance and Policy, 7(4), 375-424. 


Boyd, V., Friesen, F., Hunt, P., Hunt, S., Magoon, T., & Van Brunt, J. (1996). A summer 
retention program for students who were academically dismissed and applied for 
reinstatement (Research Report No. 13-96). College Park, MD: University of Maryland 
Counseling Center. (ERIC Document Reproduction Service No. ED405529). 


Brock, T., & LeBlanc, A. (with MacGregor, C.). (2005). Promoting student success in community 
college and beyond: The Opening Doors Demonstration. New York: MDRC. 


Brock, T., & Richburg-Hayes, L. (2006). Paying for persistence: Early results of a Louisiana 
scholarship program for low-income parents attending community college. New York: 
MDRC. 


Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked 
improvement communities in education. In Frontiers in Sociology of Education (pp. 127- 
162). Springer, Dordrecht. 


Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How 
America’s schools can get better at getting better. Cambridge, MA: Harvard Education 
Press. 


Carnevale, A., Smith, N., & Strohl, J. (2010). Help wanted: Projections of jobs and education 
requirements through 2018. Washington, DC: Georgetown University Center on Education 
and the Workforce. 


Abt Associates References [| pg. 92 


To Improve and to Prove 


Carnevale, A., Smith, N., & Strohl, J. (2013). Recovery: Job growth and education requirements 
through 2020. Washington, DC: Georgetown University Center on Education and the 
Workforce. 


Dadgar, M., & Weiss, M. J. (2012). Labor market returns to sub-baccalaureate credentials: How 
much does a community college degree or certificate pay? (CCRC Working Paper No. 45). 
New York: Community College Research Center, Columbia University. 


Darche, S., Nayar, N., & Bracco, K. R. (2009). Work-based learning in California: Opportunities 
and models for expansion. WestEd. 


Fein, D., & Hamadyk, J. (2018). Bridging the opportunity divide for low-income youth: 
Implementation and early impacts of the Year Up program (OPRE Report # 2018-65). 
Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children 
and Families, U.S. Department of Health and Human Services. 


Fishman, B. J., Penuel, W. R., Allen, A. R., Cheng, B. H., & Sabelli, N. O. R. A. (2013). Design- 
based implementation research: An emerging model for transforming the relationship of 
research and practice. National Society for the Study of Education, 112(2), 136-156. 


Fox, H. L. (2014). Achieving their goals: Implementing an individualized learning plan process to 
build student success. Champaign, IL: Office of Community College Research and 
Leadership, University of Illinois at Urbana-Champaign. 


Fuller, J., Raman, M., et al. (2017). Dismissed by degrees: How degree inflation is undermining 
U.S. competitiveness and hurting America’s middle class. Published by Accenture, Grads of 
Life, and Harvard Business School. 


Goldberger, S. (1994). Learning through work: Designing and implementing quality worksite 
learning for high school students. School-to-Work Transition Project. New York: Manpower 
Development Research Corporation. 


Goldrick-Rab, S. (2010). Challenges and opportunities for improving community college student 
success. Review of Educational Research, 80(3), 437-469. 


Gruber, E., Mandl, |., & Oberholzner, T. (2008), Learning at the workplace. In Modernising 
Vocational Education and Training (Vol. 2). Luxembourg: European Centre for the 
Development of Vocational Training. 


Gueron, J., & Rolston, H. (2013). Fighting for reliable evidence. Thousand Oaks, CA: Russell 
Sage. 


Gupta, H. (2017). The power of fully supporting community college students: The effects of the 
City University of New York’s Accelerated Study in Associate Programs after six years. New 
York: MDRC. 


Hamilton, M., & Hamilton, S. (1997). Learning well at work: Choices for quality. Washington, 
DC: National School-to-Work Opportunities Office. 


Jepsen, C., Patel, D., & Troske, K. (2010). An exploratory analysis of the relationship between 
student earnings and postsecondary retention (Discussion Paper 2010-13). Lexington, KY: 
University of Kentucky Center for Poverty Research. 


Juszkiewicz, J. (2017). Trends in community college enrollment and completion data, 2017. 
Washington, DC: American Association of Community Colleges. 


Kallison, J. M., Jr. (2017). The effects of an intensive postsecondary transition program on 
college readiness for adult learners. Adult Education Quarterly, 67(4), 302-321. 


Abt Associates References | pg. 93 


To Improve and to Prove 


Lewis, C. (2015). What is improvement science? Do we need it in education? Educational 
Researcher, 44(1), 54-61. 


Lewis, K., & Gluskin, R. (2018). Two futures: The economic case for keeping youth on track. 
New York: Measure of America, Social Science Research Council. 


Loprest, P., Spaulding, S., & Nightingale, D. S. (2019). Disconnected young adults: Increasing 
engagement and opportunity. The Russell Sage Foundation Journal of the Social Sciences, 
5(5), 221-243. 


Maguire, S., Freely, J., Clymer, C., Conway, M., & Schwartz, D. (2010). Tuning in to local labor 
markets: Findings from the Sectoral Employment Impact Study. Philadelphia, PA: 
Public/Private Ventures. 


Mahlberg, J. (2015). Formative self-assessment college classes improves self-regulation and 
retention in first/second year community college students. Community College Journal of 
Research and Practice, 39(8), 772-783. 


Maroto, M. E., Snelling, A., & Linck, H. (2015). Food insecurity among community college 
students: Prevalence and association with grade point average. Community College Journal 
of Research and Practice, 39(6), 515-526. 


Maynard, R., Baelen, R., Souvanna, P., Fein, D., & Shivji, A. (2018). Final evaluation report for 
Year Up’s Professional Training Corps program in Philadelphia. Rockville, MD: Abt 
Associates. 


National Research Council. (2014). Investing in the health and well-being of young adults. 
Washington, DC: The National Academies Press. 


National Student Clearinghouse Research Center. (2018). Enrollment coverage workbook. 
Database. https://nscresearchcenter.org/wp-content/uploads/NSC_COVERAGE.xIsx 


National Student Clearinghouse Research Center. (2019). Yearly success and progress rates 
(fall 2012 entering cohort). Snapshot Report. https://nscresearchcenter.org/wp- 


content/uploads/SnapshotReport34. pdf 


Organisation for Economic Development (OED). (2010). Learning for jobs. Synthesis report of 
the OECD reviews of vocational education and training. Paris, France: ???????. 


Orr, L. L. (1999). Social experiments: Evaluating public programs with experimental methods. 
Thousand Oaks, CA: Sage. 


Roder, A., & Elliott, M. (2019). Nine year gains: Project QUEST’s continuing impact. New York: 
Economic Mobility Corporation. 


Roder, A., & Elliot, M. (2014). Sustained gains: Year Up’s continued impacts on young adults’ 
earnings. New York: Economic Mobility Corporation. 


Rogers-Chapman, M. F., & Darling-Hammond, L. (2013). Preparing 21st century citizens: The 
role of work-based learning in linked learning. Stanford, CA: Stanford Center for Opportunity 
Policy in Education. 


Rolston, H., Copson, E., & Gardiner, K. (2017). Valley Initiative for Development and 
Advancement: Implementation and early impact report (OPRE Report # 2017-83). 
Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children 
and Families, U.S. Department of Health and Human Services. 


Abt Associates References 1 pg. 94 


To Improve and to Prove 


Ross, M., K. A. Moore, K. Murphy, N. Bateman, A. DeMand, and V. Sacks. 2018. Pathways to 
High-quality Jobs for Young Adults. Washington, DC: Brookings Institution. 


Russell, J. L., Bryk, A. S., Dolle, J., Gomez, L. M., LeMahieu, P., & Grunow, A. (2017). A 


framework for the initiation of networked improvement communities. Teachers College 
Record, 119(7), 1-36. 


Schaberg, K. (2017). Can sector strategies promote longer-term effects? New York: MDRC. 


Scrivener, S., & Weiss, M. J. (2009). More guidance, better results? Three-year effects of an 
enhanced student services program at two community colleges. New York: MDRC. 


Scrivener, S., Sommo, C, & Collado, H. (2009). Getting back on track. New York: MDRC. 
Sokovic, M., Pavletic, D., & Pipan, K. K. (2010). Quality improvement methodologies-PDCA 
cycle, RADAR matrix, DMAIC, and DFSS. Journal of Achievements in Materials and 

Manufacturing Engineering, 43(1), 476-483. 

Sowers, N., & Yamada, H. (2015). Community college pathways: 2013-2014 descriptive 
report. Stanford, CA: Carnegie Foundation for the Advancement of Teaching. Accessed 5- 
19-19 at: https://files.eric.ed.gov/fulltext/ED554591 .pdf 

Stinebrickner, R., & Stinebrickner, T. R. (2003). Working during school and academic 
performance. Journal of Labor Economics, 21(2), 473-491. 

Van Noy, M., Trimble, M., Jenkins, D., Barnett, E., & Wachen, J. (2016). Guided pathways to 
careers: Four dimensions of structure in community college career-technical 
programs. Community College Review, 44(4), 263-285. 

Weiss, M. J., Ratledge, A., Sommo, C., & Gupta, H. (2019). Supporting community college 


students from start to degree completion: Long-term evidence from a randomized trial of 
CUNY’s ASAP. American Economic Journal: Applied Economics, 11(3), 253-297. 


Weiss, M. J., Visher, M. G., & Wathington, H. (2010). Learning communities for students in 
developmental reading: An impact study at Hillsborough Community College. National 
Center for Postsecondary Research. 


Weissman, E., Cullinan, D., Cerna, O., Safran, S., & Richman, P. (2012). Learning communities 
for students in developmental English: Impact studies at Merced College and the 
Community College of Baltimore County. New York: MDRC. 


Year Up. (2018). Mini-experiment: Alternative coaching resources. Mimeo. Boston, MA: Author. 


Abt Associates References [J pg. 95 


